[ENG-10788] add nodes and preprints files reinding on account merging

[mkovalua]

Ticket

• https://openscience.atlassian.net/browse/ENG-10788

Purpose

User merge does not seem to be working for files

Changes

Adding nodes and preprints files reinding on account merging

Also some updates for SHARE is good to have

CenterForOpenScience/SHARE#892

Side Effects

File indexing does not work for local setup, so maybe it is needed add something else because something may be missed because of a bit blame fix

QE Notes

CE Notes

Documentation

[antkryt]

Just a small import fix

[antkryt]

LGTM

[cslzchen]

The fix looks good but I have two notes:

• Is this feature testable by unit tests? If so, please update/add.
• Are there any possible impact to the celery queue if too many update_share(file), could it crowd/block the celery queue?

[mkovalua]

The fix looks good but I have two notes:

• Is this feature testable by unit tests? If so, please update/add.
• Are there any possible impact to the celery queue if too many update_share(file), could it crowd/block the celery queue?

Hi @cslzchen

1. Have updated unittests

2. Hard to say from my side (had problems with local files share indexing as shown in PR description), I suppose that it may be possible. If yes - maybe the solution is to create low priority tasks for files and setup several workers and it may be helpful. Maybe @mfraezz and @aaxelb have ideas. Thanks

[aaxelb]

• Are there any possible impact to the celery queue if too many update_share(file), could it crowd/block the celery queue?

2. Hard to say from my side (had problems with local files share indexing as shown in PR description), I suppose that it may be possible. If yes - maybe the solution is to create low priority tasks for files and setup several workers and it may be helpful. Maybe mfraezz and aaxelb have ideas. Thanks

+1 for putting bulk tasks like these on a lower-priority queue -- currently always high priority

over in SHARE there's task routing to prioritize certain tasks called with urgent=True -- could do similar in OSF with (or replacing) the is_backfill kwarg to task__update_share (which indirectly decides urgency in SHARE already), and allowing something like update_share(obj, urgent=False) for updates not directly spurred by user interaction

[cslzchen]

The approach should work 👍 .

I'd like to see if we can improve and generalize it so that this can be used by other tasks and that it doesn't conflict with other tasks.

Afterwards, make sure we have good docstring and unit tests.

[cslzchen]

Add docstring, mentioning which queue we are using, the default behavior and why we should use one over the other.

[cslzchen]

Ditto, docstring

[cslzchen]

Ditto, docstring

[cslzchen]

Can you take a further look at https://docs.celeryq.dev/en/latest/userguide/routing.html#routers to see if this is the proper way?

In addition, my understanding is the args and kwargs comes from the task. It's very likely other tasks may use urgent for a different purpose or even with different type.

• Please check if we have any tasks that uses previously uses urgent?
• Even if not, should we provide a way to avoid urgent being used in the future. My though is use a more verbose/explicit name. e.g.

def update_share(..., target_queue=None)
   ...

update_share(..., target_queque=CeleryConfig.task_low_queue)

• In addition, make sure CeleryRouter verifies the queue is valid if manually selected.