#761 Fix handling of special characters in column names by adding backtick escaping in schema corrections.

[yruslan]

Closes #761

Test Evidence

The change confirmed fixed the ingestion from the original dataset that contain parenthesis in decimal field names.

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of corrected decimal fields in generated Spark SQL schemas by safely quoting field names, reducing issues with special characters or reserved words.

[coderabbitai]

Warning

Review limit reached

@yruslan, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 40 minutes and 1 second. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4f76e163-64a0-427b-a0e7-c648cff3d549

📥 Commits

Reviewing files that changed from the base of the PR and between e265460 and e5fafcf.

📒 Files selected for processing (2)

• pramen/core/src/main/scala/za/co/absa/pramen/core/utils/JdbcSparkUtils.scala
• pramen/core/src/test/scala/za/co/absa/pramen/core/tests/utils/JdbcSparkUtilsSuite.scala

Walkthrough

In getCorrectedDecimalsSchema within JdbcSparkUtils.scala, field names are now wrapped in backticks when constructing Spark SQL custom schema fragment strings. This change applies to all four decimal correction cases: integer, long, decimal(38, 18), and decimal(fixedPrecision, scale).

Changes

Backtick quoting for decimal schema field names

Layer / File(s)	Summary
Backtick-quote field names in schema fragments pramen/core/src/main/scala/za/co/absa/pramen/core/utils/JdbcSparkUtils.scala	All four schema string cases in getCorrectedDecimalsSchema (lines 280–291) now emit the corrected field name enclosed in backticks, replacing the prior unquoted interpolation.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

A field named with parens once caused Spark to frown,
But backticks arrived and saved the whole town.
`PaidAmount (Incl VAT)` now parses just right,
The rabbit wrapped names in quotes, snug and tight.
🐇✨ No more parse exceptions tonight!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: backtick escaping for special-character column names in JDBC schema corrections.
Linked Issues check	✅ Passed	The schema correction now quotes corrected decimal column names with backticks while keeping precision/scale handling intact.
Out of Scope Changes check	✅ Passed	The diff is narrowly focused on JDBC schema quoting and decimal correction, with no unrelated changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch bugfix/761-decimal-corrections-with-parenthesis

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@pramen/core/src/main/scala/za/co/absa/pramen/core/utils/JdbcSparkUtils.scala`:
- Around line 280-291: Update JdbcSparkUtilsSuite to match the new quoted-schema
behavior from JdbcSparkUtils; the current assertions still expect unquoted field
fragments, so revise the tests to assert backtick-quoted column names and
aligned type strings wherever the suite checks the schema output. Also add a
regression test covering a name like PaidAmount (Incl VAT) to verify the quoted
contract is preserved end-to-end when generating the corrected schema.
- Around line 280-291: Escape any backticks inside field.name before building
the customSchema strings in JdbcSparkUtils.correctSchema so quoted identifiers
remain valid Spark SQL; update the schema-building cases that append `newSchema`
for the DecimalType/other field branches to use doubled backticks within the
surrounding backticks, and adjust JdbcSparkUtilsSuite expectations to assert the
escaped quoted output.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 252a5c23-eee3-4902-8ec6-857b9ae37715

📥 Commits

Reviewing files that changed from the base of the PR and between 6973716 and e265460.

📒 Files selected for processing (1)

• pramen/core/src/main/scala/za/co/absa/pramen/core/utils/JdbcSparkUtils.scala

[github-actions]

Unit Test Coverage

Overall Project	76.91%	🍏
Files changed	100%	🍏

Module	Coverage
pramen:core Jacoco Report	77.9%	🍏

Files

Module	File	Coverage
pramen:core Jacoco Report	JdbcSparkUtils.scala	91.7%	🍏