Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling multiple consecutive event sources #146

Open
nathanwbrei opened this issue Aug 18, 2022 · 4 comments
Open

Handling multiple consecutive event sources #146

nathanwbrei opened this issue Aug 18, 2022 · 4 comments

Comments

@nathanwbrei
Copy link
Collaborator

In JANA1, the user could specify a list of event sources. Each event source would be opened, fully processed, and closed, and then JANA1 would move on to the next event source.

In JANA2, currently, the event sources are run in parallel, which is generally not the desired behavior. This is a straightforward change to JEventSourceArrow and JTopologyBuilder.

@nathanwbrei nathanwbrei added this to the v2.0.8: Support for EICrecon milestone Aug 18, 2022
@nathanwbrei
Copy link
Collaborator Author

@faustus123 What is the desired interaction between nskip/nevents and multiple event sources? Our options are:

  1. Give people the option to set nskip/nevents on a per-JEventSource basis, e.g. by specifying something like jana myinput.root:10:100 to have JANA skip the first 10 events and process the next 100.

  2. Apply the global jana:nskip and jana:nevents to each source (unless the user already overrode the values for that particular source). So jana -Pjana:nskip=10 -Pjana:nevents=100 file1.root file2.root will skip the first 10 from file1, process the next 100 from file 1, close file1 and open file2, skip the first 10 from file2, and process the next 100 from file2. This is what we currently do

  3. Apply the global jana:nskip and jana:nevents to the stream of all JEvents emitted from all sources, e.g. jana -Pjana:nskip=10 -Pjana:nevents=100 file1.root file2.root file3.root (where each file contains 50 events) will drop the first 10 events from file1.root, process the next 40 events from file1.root, process all 50 events from file2.root, and process the first 10 events from file3.root before exiting.

@faustus123
Copy link
Collaborator

Good question. I would guess 3. would cover the most common use case. However, this will be very annoying for those who actually want use case 1 or 2. I'm not sure how hard 1 is, but it would mostly cover case 2 as well. Would it be hard to implement 1 and 3? You could throw an exception if inconsistent values are specified.

If it seems very complicated, I would probably prefer 3. and then defer the more complex scenarios until a user requests them.

@nathanwbrei
Copy link
Collaborator Author

I'll give it a shot, but put it in another pull request

@nathanwbrei nathanwbrei removed this from the v2.0.8: Support for EICrecon milestone Sep 22, 2023
@nathanwbrei
Copy link
Collaborator Author

Event sources have run one after another since #176. What has been missing thus far is sensible handling of nevents/nskip. The reason nevents/nskip is weird is because it shouldn't live at the JEventSource level at all, but rather the JEventSourceArrow. Furthermore, nevents/nskip shouldn't be reading and immediately discarding data; it should be jumping to the nskip'th event. Right now, JEventSource doesn't support random file access. It is easy to add, though: We just need bool Seek(uint64_t entry_nr) and uint64_t GetEntryCount() callbacks, and have the default implementation return false, indicating that the seek failed.

@nathanwbrei nathanwbrei changed the title Running event sources one after another Handling multiple consecutive event sources Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants