self-hosted v4.4.6 unable to see any logger event information

[insanity54]

Hi I'm getting started with trigger.dev and I've run into a very annoying problem-- I can't see any logs!

This is a task that successfully ran to completion, but there is no event data, and no logs!

There should be log SPAM, because this is part of the code.

export const audioAnalysisTask = task({
  id: "audio-analysis",
  maxDuration: 360,
  run: async (
    payload: { audioId: string; twitchLogin: string },
    { ctx },
  ) => {
    const { audioId, twitchLogin } = payload;
    const streamSegmentedDuration = ctx.environment.type === 'PRODUCTION' ? '00:05:00' : '00:00:30';
    const baseUrl = process.env.TAILNET_DNS_NAME
      ? `https://${process.env.TAILNET_DNS_NAME}`
      : process.env.PUBLIC_ORIGIN ?? "http://localhost:3000";
    const workerToken = process.env.WORKER_TOKEN;

    console.log(`hello this is a console.log. the baseUrl is`, baseUrl);
    logger.log(`hello this is logger.log and i'll shove some metadata in here.`, { "derp": true });
    logger.info(`hello this is logger.info and I have no metadata`);
    logger.debug('hello this is debug');
    logger.warn('helo this is logger.warn');

But nope, it's all empty, except for that console.log which does show in the terminal, but not in the webUI.

How about the (alpha) logs page?

Only errors and traces. What in the world? Where are the INFO, and LOG levels?

I got the idea to connect to an official trigger.dev cloud instance to see if the problem exists there too. It doesn't! I see logs on official cloud!

I guess this is how it's supposed to look.

I figure there's something wrong with my self-hosted config. The problem? I don't know WHAT. What part? What service? What env var? I have no idea what setting would be causing this.

I dove through logs from the various docker-compose services and I didn't see anything that stuck out.

Any clues as to what could be causing this?

[yudin-s]

For self-hosted Trigger, this usually means the run completed but event/log ingestion is not healthy (UI can show run status while logs/events stay empty).\n\nFast checks in order:\n1. Verify all Trigger services are on the same version (v4.6.6) and healthy (web, worker, and any event/log storage dependencies).\n2. Open browser devtools on the run page and check failing requests for run events/logs (4xx/5xx gives the exact missing component).\n3. Check worker/container env for required storage/queue vars and connectivity (DB/redis/object store depending on your setup).\n4. Trigger a tiny task with only logger.info("hello") and console.log("hello") to see if any log path is working.\n\nYour java.net.SocketException: Network is unreachable in logs is a strong signal that at least one internal fetch path is failing from inside the container network. Fixing that network path usually restores event/log visibility too.\n\nIf you can share one failing request from the run page network tab (URL + status), we can pinpoint the exact component. If this solved it, please mark the answer so others can find it faster.

[insanity54]

Brother you waste my time with your loose cannon AI agent

[insanity54]

Ok so I figured it out, with many thanks to comments by the devs on the Discord. Lots of people struggle with this same problem, so I'll reference those discussions here.

• missing logs by mikirejf. They worked around the issue by downgrading to beta.26
• v4 of trigger dev not logging by rafaelsene01. I don't think they solved their issue.
• logs are not reliably coming through by juienvanbeveren. The conversation suggested ensuring clickhouse was properly configured.

  trigger.dev/apps/webapp/app/env.server.ts

  Lines 1137 to 1156 in 892bed8

  	EVENTS_CLICKHOUSE_URL: z
  	.string()
  	.optional()
  	.transform((v) => v ?? process.env.CLICKHOUSE_URL),
  	EVENTS_CLICKHOUSE_KEEP_ALIVE_ENABLED: z.string().default("1"),
  	EVENTS_CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS: z.coerce.number().int().optional(),
  	EVENTS_CLICKHOUSE_MAX_OPEN_CONNECTIONS: z.coerce.number().int().default(10),
  	EVENTS_CLICKHOUSE_LOG_LEVEL: z.enum(["log", "error", "warn", "info", "debug"]).default("info"),
  	EVENTS_CLICKHOUSE_COMPRESSION_REQUEST: z.string().default("1"),
  	EVENTS_CLICKHOUSE_BATCH_SIZE: z.coerce.number().int().default(1000),
  	EVENTS_CLICKHOUSE_FLUSH_INTERVAL_MS: z.coerce.number().int().default(1000),
  	EVENTS_CLICKHOUSE_INSERT_STRATEGY: z.enum(["insert", "insert_async"]).default("insert"),
  	EVENTS_CLICKHOUSE_WAIT_FOR_ASYNC_INSERT: z.string().default("1"),
  	EVENTS_CLICKHOUSE_ASYNC_INSERT_MAX_DATA_SIZE: z.coerce.number().int().default(10485760),
  	EVENTS_CLICKHOUSE_ASYNC_INSERT_BUSY_TIMEOUT_MS: z.coerce.number().int().default(5000),
  	EVENT_REPOSITORY_CLICKHOUSE_ROLLOUT_PERCENT: z.coerce.number().optional(),
  	EVENT_REPOSITORY_DEFAULT_STORE: z.enum(["postgres", "clickhouse"]).default("postgres"),
  	EVENTS_CLICKHOUSE_MAX_TRACE_SUMMARY_VIEW_COUNT: z.coerce.number().int().default(25_000),
  	EVENTS_CLICKHOUSE_MAX_TRACE_DETAILED_SUMMARY_VIEW_COUNT: z.coerce.number().int().default(5_000),
  	EVENTS_CLICKHOUSE_MAX_LIVE_RELOADING_SETTING: z.coerce.number().int().default(2000),

• log messages don't show up in the task runs by Emad Ibrahim. Trigger.dev staff said, "Logs for dev get sent to the API server and use open telemetry. I'd try do some logs from a dev run and look at the server logs and see what you get. Something is misconfigured somewhere. Compare our self-hosting guide with your setup too."
• detailed logs missing by Leoodaviid Staff said, "- Runs can execute correctly but logs don’t come through. That’s usually when your OTEL collector isn’t setup correctly
  Which means the logs never make it from your running machine into the db"
• Can not see any logs on trigger run by Phillip. I don't think they solved their issue.
• No logs or details after upgrading the machine to large-1x by praneethpike. (This was on the cloud version of trigger.dev.) They resolved their problem by switching their region from us-east-1 to europe
• Logs only shows root task by blekekok. Erin said this is usually caused by OTEL_EXPORTER_OTLP_ENDPOINT defined incorrectly. Nick said to check this env var which in the example is the FQDN URL (eg: https://trigger.sbtp.xyz/), not the local-to-docker http://trigger:3000/otel like I have my set as.

The biggest tell for me was this .env var configuration example. Notice how APP_ORIGIN and OTEL_EXPORTER_OTLP_ENDPOINT are the same origin (this challenged my assumption that they should be different.)

trigger.dev/hosting/docker/.env.example

Line 46 in eefb96c

DEV_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8030/otel

Long story short, my DEV_OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_ENDPOINT were wrong. I was incorrectly assuming that tasks run on Coolify with the rest of the docker containers, so I set those vars to http://trigger:3000/otel. In reality, (in development environment) the tasks run on my local computer, outside of docker, which means that that URL won't resolve. I wish there was a 404 in the logs somewhere to indicate that, but I never found one.

Anyway, once I set DEV_OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_ENDPOINT to my self-hosted instance at https://trigger.sbtp.xyz/otel, then restarted everything including the bunx trigger.dev@latest dev --api-url https://trigger.sbtp.xyz --profile self-hosted, I was able to start seeing the logs and other open telemetry data.