Re-assess CI build

[greggyb]

Some issues we've identified:

• superfluous builds and extra doc sites during PR review (crowdin & adding reviewers can kick off extra builds we don't need; Nedas has more detail)
• run scripts have some candidates to add to CI
  • ./run scripts check: for linting+formatting on changes to .py and .sh files
  • ./run doctest: for markdown file changes. This command automatically discovers files that have annotated C# scripts (ignores most docs automatically); this is safe and opt-in per doc file
  • ./run check-links: maybe not ready for primetime in its current form. We should talk through it some more (discussion below)
• Target actions to specific paths/globs so we don't rebuild when we don't need to

For link checking, we have two categories of links: intra-doc links (to other docs we have authored in the repo) and external links. Intra-doc links also subdivide to old-path redirects and links to live paths. External links can be down transiently, and we have no control. Right now the check-links script makes a decision between these: intra-doc links are considered errors and external links are considered warnings.

With regard to re-directs: we validate internal links in two ways:

1. Against the built docs site on local disk. This requires no network.
2. For the subset of old-style links, we hit the live docs site to ensure we get correct redirect logic. This requires network.

External sources obviously require network.

Perhaps we adopt a threshold for external links, rather than any single dead link failing a build? I have seen some links that were dead come back alive while working on this script.

A recent scan of links is in #359

[Metallist1]

Regarding builds: Yes that is on my backlog.
Regarding CI improvements: I noted down both linting and doctest.
Regarding links, the only links we should fail on is the intra-docs links. External links are out of our control - we should have a warning but never fail our builds due to them. My reasoning is that anything could knock out an external link for a short period or a long period of time. (DDOS attacks, Cloudflare being down or even our own network being slow or down...). So I feel that external link checkup should not be part of the CI or if it is - it should be only warnings that would not fail the build.

Regarding targeting only relevant parts : This is more relevant for local development than CI/CD. Mainly as we do want to build the entire app when we deploy. It is to ensure that the entire app works when we merge it with main and the benefit of shortening deployment time is not worth compared to the complexity of the solution.

[greggyb]

My thinking on external links was that we might do a threshold. If there are >N broken external links, that merits some attention by a human. We could let this be an overridable check.

Or if we want to be a bit more focused, we could check external links only in the modified files (this would require a bit of changing to the check script for the granularity, but doable easily). Then we could automatically add a comment to the PR for any broken external links in the changed docs. This puts it on the PR author or a reviewer to check the links. Then it's up to a human to resolve that auto-comment, but doesn't hard-block the PR.

I had thought maybe we don't build the whole app for helper script-only changes. But with even a little extra thought, I see how I'm wrong there. If we end up depending on a script in CI, then changes to that script do affect the site build. Brain fart on this one q:

[Metallist1]

I would still disagree on providing a threshold for failure on external links. I would like them to be warning at most. The issue here is that we should account for things in our control when we are building. Internal links failing and causing the build to fail is fair, but external links should be at most warnings. The reason for it is that:
a) We do not control the content on external links or the accuracy of the content. So we have no control over correctness.
b) We do not control the availability of external links.
c) External links would become a blocker that would need to be addressed before we can push updates to the docs. Example: If microsoft documentation goes down today for 24 hours - it stops all PRs from being merged for our docs site. Why should we stop our development because microsoft is unable to maintain their own sites ?
d) Maintenance of historical records in release history. If a link goes down for Tabular Editor 3.22.1 - how do we tackle it ? Do we remove it ? Do we a find a replacement ? Is it relevant enough to block our progress until it is fixed ? Is it worth the effort in fixing it ? Ect.

So to me, it only makes sense to have warnings but no blockers.