Skip to content

Fixed UDF jar metadata handling in UDFInfo when multiple UDFs share the same jar#17732

Open
Caideyipi wants to merge 2 commits into
masterfrom
udf-fix
Open

Fixed UDF jar metadata handling in UDFInfo when multiple UDFs share the same jar#17732
Caideyipi wants to merge 2 commits into
masterfrom
udf-fix

Conversation

@Caideyipi
Copy link
Copy Markdown
Collaborator

Description

This PR fixes UDF jar metadata handling in UDFInfo when multiple UDFs share the same jar.

Background

UDFInfo previously tracked uploaded jars only with jarName -> md5. When a UDF was dropped, the jar metadata was
removed immediately. This breaks the case where multiple UDFs reference the same jar:

  • dropping one UDF can make ConfigNode think the shared jar no longer exists
  • later validation may no longer detect conflicting MD5 values for the same jar name
  • after snapshot load, jar metadata is restored but shared-jar reference state is not rebuilt, so the same issue can
    reappear after restart

Changes

This PR introduces reference counting for shared UDF jars in UDFInfo.

  • add existedJarToReferenceCount to track how many UDFs are using each jar
  • update UDF creation flow to increase jar reference count instead of only recording MD5
  • update UDF drop flow to remove jar metadata only when the last reference is removed
  • rebuild jar metadata and reference counts from the UDF table after loading a snapshot

This keeps shared jar metadata consistent across normal create/drop operations and snapshot recovery.

Tests

Updated UDFInfoTest to cover the shared-jar cases:

  • dropping one UDF does not remove jar metadata if another UDF still references the same jar
  • validation still rejects the same jar name with a different MD5 after one reference is dropped
  • snapshot load rebuilds shared-jar metadata correctly, and subsequent drop behavior remains correct

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

@sonarqubecloud
Copy link
Copy Markdown

@codecov
Copy link
Copy Markdown

codecov Bot commented May 21, 2026

Codecov Report

❌ Patch coverage is 87.50000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.56%. Comparing base (7563ac8) to head (2c62c39).

Files with missing lines Patch % Lines
...g/apache/iotdb/confignode/persistence/UDFInfo.java 87.50% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17732      +/-   ##
============================================
+ Coverage     40.55%   40.56%   +0.01%     
- Complexity     2574     2575       +1     
============================================
  Files          5179     5179              
  Lines        349896   349918      +22     
  Branches      44727    44731       +4     
============================================
+ Hits         141890   141954      +64     
+ Misses       208006   207964      -42     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Member

@luoluoyuyu luoluoyuyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review 总结

整体方向正确:用引用计数修复「多 UDF 共享同一 jar 时 drop 一个误删 jar 元数据」的问题,rebuildJarMetadataFromUDFTable() 也能在 snapshot 恢复后重建一致性。UT 覆盖了共享 jar 与 snapshot 场景,质量不错。

建议在合入前处理下面 2 点(见行内评论),其中 MD5 一致性建议在 addJarReference 内防御性校验。

}

private void addJarReference(String jarName, String jarMD5) {
existedJarToMD5.putIfAbsent(jarName, jarMD5);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

putIfAbsent 只会在 jar 名已存在时保留第一次写入的 MD5,但仍会执行 merge 增加引用计数。

当前 validate()addUDFInTable 之前会拦截「同名 jar、不同 MD5」,因此正常路径是安全的。但若未来有代码路径绕过 validate 直接调用 addJarReference,会出现 refCount 与 MD5 不一致。

建议:在 addJarReference 内增加防御性检查,例如:

final String existing = existedJarToMD5.get(jarName);
if (existing != null && !existing.equals(jarMD5)) {
  throw new IoTDBRuntimeException(...);
}
existedJarToMD5.putIfAbsent(jarName, jarMD5);

这样与 validate() 形成双保险。

deserializeExistedJarToMD5(fileInputStream);

udfTable.deserializeUDFTable(fileInputStream);
rebuildJarMetadataFromUDFTable();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

processLoadSnapshotdeserializeExistedJarToMD5rebuildJarMetadataFromUDFTable(),后者会 clear 并完全从 udfTable 重建 jar 元数据,使得反序列化的 existedJarToMD5 内容被丢弃。

若这是有意为之(以 udfTable 为唯一真相源),建议在方法上加一行注释说明,避免后续维护者误以为需要同时保留 snapshot 里的 jar map。

另:旧版本 snapshot 若 existedJarToMD5 与 udfTable 不一致,rebuild 能自动修复,这是加分项。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants