Add in-memory caching for directory reads

[zapdos26]

This pull request introduces an in-memory cache for directory reads in the getFilesInDirectory function to optimize performance by avoiding redundant reads during a single module command invocation. Additionally, a new test has been added to verify that the cache returns consistent results for repeated queries.

Caching improvements:

• Added an in-memory cache to the getFilesInDirectory function in tcl/main.tcl.in, wrapping the original implementation and storing results for repeated directory queries within a single invocation.

Testing enhancements:

• Introduced a new test in testsuite/modules.90-avail/025-memcache.exp to confirm that consecutive avail queries on the same module path return consistent results, validating the effectiveness of the cache.

[smprather]

Adding comments I got from Codex.

• For PR Add in-memory caching for directory reads #613: I would leave one review comment saying the added test does not validate cache reuse because each assertion is a separate process.

Details

• Medium: the new test does not appear to exercise the cache it is meant to validate. In the PR diff, testsuite/modules.90-avail/025-memcache.exp uses separate testouterr_cmd calls. In this repo, modules-main/
  testsuite/config/base-config.exp:1093 calls modules-main/testsuite/modules.00-init/006-procs.exp:27, which goes through modules-main/testsuite/config/unix.exp:53 and spawns a fresh modulecmd process each
  time. So the per-process in-memory cache is reset between assertions. The test checks output stability, not cache reuse.
• Low: the cache key in the PR is built as a string concat, "$dir:$fetch_dotversion" in tcl/main.tcl.in:637 from the PR diff. It would be cleaner to use a Tcl list key or nested array to avoid ad hoc key
  encoding.

[xdelaruelle]

Thanks @smprather for your comment and @zapdos26 for your change proposal.

@zapdos26 could you to elaborate a bit on the redundant reads you mention.

getFilesInDirectory is called either through findModules or findModulesInCacheFile. These two procedures have a memory cache in place already.

The cache behavior of findModules is tested in testsuite/modules.20-locate/090-memcache.exp.

So from my understanding redundant reads should not be observed with existing code.

[smprather]

I (via Codex) did some work on it. Unfortunately, I close on a new house tomorrow, and moving 5 miles down the road becomes my top-prio for a while. GPT fixed everything to make it CI-clean, including improving some false-failing tests (loosening some regex's). If there's no rush, I can provide the results as time permits.

[smprather]

Codex said the performance improvement was minor, but it's there. @xdelaruelle : I'll feed in your question and see what it thinks.