I guess the next natural step would be to show the timestamped comments as annotations just like Viddler does it.
A spot of rain on the parade is that the geo IP redirect in YouTube breaks timestamped links as you get sent from www. to uk. (for example here in the UK as found by Simon Willison). This should be easy to fix though, please?