Virtual timelines are very easy. You just need to:
1. Open all the segments (files), each one on its own context, so you can read A/V/S samples from them given a local timecode
2. Build a table that maps the global (timelined) timecode periods to file/local-timecode-offset pairs
Then, to get data at any given time, you look up the table and extract the data from where it says you should.
So conceptually it's not that hard. It's just laborious (parsing the chapters; searching, opening and parsing the files; adding sanity checks such as checking track-per-track compatibility...)