This blog forms part of the http://performanceGuru.com website which focuses on performance tuning UNIX-like Operating Systems (including Linux).
I have spent quite a bit of time recently looking at Dtrace and Systemtap. The idea is not new, research seems to indicate that the first attempt as this type of probe insertion to allow profiling of code may have appeared in OS2. Both of these tools are really opening the lid on the Operating System, from even simple requirements (who is wrtiting what to disk, the elusive dtop command) through to complex tasks (what causing all those cross calls).
Both allow the horizontal and vertical views that IT always seems to provide. The vertical view is the deterministic profiling option, in which is is possible to follow a single thread of execution, taking sampels at function boundaries, to allow for that thread to be highly optimised. On the horizontal side is the ability to take systemic views which are more akin to statistical profiling.
However there are still a lot of deployed instances that are neither Dtrace or Systemtap capable. For this reason it would seem to be useful to allow for a newer tracing capable O/S to be booted and 'unioned' with an existing Production system to allow for an application to be profiled outside of core Production hours without upgrading the O/S.
For Solaris it would seem to make sense to boot some kind of network avaialbe Solaris 10 instance that maps in the existing root read-only into a zone and then uses this configure the Solaris 10 instance and to provide the local libraries for the application.
For Linux it w0uld seem sensible to do something similar with initrd and PXE. This would allow for an application to be booted using the existing read-only root mounted onto the initrd image.
The two major issues would seem to be versions of the ABI. Solaris should be fairly immune to this as Sun gurantee total backwards compatability and guarantee to fix any issues as bugs. In the RedHat world the big divider would seem to be the threading model, however these kinds of problems can be overcome using LD_ASSUME_KERNEL if necessary. Usually RedHat seem to be very careful with the ABI and things, except for JVMs which seem to play with the stack, etc for efficiency/performance reasons, so for most applications they should work.
Maybe time to do some further research into this. If and when I get time to explore this idea I will post an update.