Skip to content

feat(io): Vended credential refresh for S3#3351

Open
gabeiglio wants to merge 3 commits into
apache:mainfrom
gabeiglio:vended-credential-refresh
Open

feat(io): Vended credential refresh for S3#3351
gabeiglio wants to merge 3 commits into
apache:mainfrom
gabeiglio:vended-credential-refresh

Conversation

@gabeiglio
Copy link
Copy Markdown
Contributor

@gabeiglio gabeiglio commented May 11, 2026

Rationale for this change

When doing long writes to a table the credentials vended with the last load table can be expired if the commit takes too long.

Here as same as Java, we proactively refresh the s3 vended credentials if they are 5 minutes away or less from expiration.

Are these changes tested?

Yes, added tests to tests/io/test_io.py to test all the logic for when its needed to call the vended credentials endpoint from the file io

Are there any user-facing changes?

Yes, users can now add the following properties to allow automatic vended credential refresh to take place

client.refresh-credentials-endpoint -> endpoint to where to call IRC
client.refresh-credentials-enabled -> Refresh credential enabled set to "true"
uri -> the catalog uri

@Fokko Fokko self-requested a review May 15, 2026 20:21
Copy link
Copy Markdown
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Automatically refreshing storage credentials is very useful.

WDYT about making this a catalog responsibility? The refresh endpoint is a catalog endpoint, so instead of passing session down to the individual FileIO layer, we could handle credential refresh in the catalog itself.

I'm thinking something like how OAuth2AuthManager operates today, it handles the auth token refresh responsibility. We could have something like CredentialProvider that the catalog creates and passes to FileIO, where FileIO just calls get_credentials().

That way the IO layer stays decoupled from REST/session concerns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants