ENT-14140: psql_wrapper.sh: retry psql commands on transient failures

[larsewi]

Observed a race condition in CI where bundle agent superhub_schema interacts with postgres shortly after service restart.

03:12:04 systemd: Stopping CFEngine Enterprise PostgreSQL Database...
03:12:04 systemd: Started CFEngine Enterprise PostgreSQL Database.
03:12:04 cf-agent: Executing ... psql_wrapper.sh cfdb select superhub_schema(...)
03:12:05 cf-agent: returned code '2' defined as promise failed

Fixed by gating superhub_schema, ensure_feeders, and imported_data on a persistent class set by the cf-postgres restart.

Ticket: ENT-14140

[craigcomstock]

This doesn't feel quite right to me. It would seem we need more of a sequence of actions and not a class/gate situation. We need the restart to finish and then run superhub_schema(). With this solution superhub_schema() would be run at next agent interval, which is OK but maybe not ideal. Could we instead of gating on recent restart gate on postgresql up and ready in hopes that superhub_schema() could run in the same agent run?

[nickanderson]

I don't think this is a great fix. a 1 minute persistence is a bit strange. 4 or 5 minutes make more sense on the default periodic scale but still I would try to think of a different way to handle this.

[larsewi]

2 if the connection to the server went bad and the session was not interactive

• https://www.postgresql.org/docs/current/app-psql.html

@craigcomstock, @nickanderson what if we have the psql_wrapper.sh retry in case of return code 2 ?

[nickanderson]

2 if the connection to the server went bad and the session was not interactive

• https://www.postgresql.org/docs/current/app-psql.html

@craigcomstock, @nickanderson what if we have the psql_wrapper.sh retry in case of return code 2 ?

yeah I think that would be better.

[larsewi]

2 if the connection to the server went bad and the session was not interactive

• https://www.postgresql.org/docs/current/app-psql.html

@craigcomstock, @nickanderson what if we have the psql_wrapper.sh retry in case of return code 2 ?

yeah I think that would be better.

The only thing @nickanderson is that it will cause the agent to hang while it bootstraps. Or should it perhaps run these commands in the background?

[nickanderson]

The only thing @nickanderson is that it will cause the agent to hang while it bootstraps. Or should it perhaps run these commands in the background?

Hang permanently, or just be slow? Permanent hang needs to be avoided. Just reading the code there it looks like it might hang for up to 30 seconds while re-trying. Also, this hang would be limited to the hub bootstrapping to itself, is that right?