Both runs have been done on a single 120GB 7200rpm IDE disk. Command used: iostat -x 5 (extended statistics, sample every 5 seconds).
On the first picture you can see iostat statistics during bonnie run, writing with putc phase. First thing you'll notice is that disk is not fully utilized (%busy less than 100%). Reason for that is that writing with putc() is effectively CPU bound. You can also see that write is sequential in nature (lots of write merges - mgw/s, and big average size of requests - elevator in the kernel is doing its job). Disk service time is slightly more than 10ms which is OK for such a big requests. Wait time is a fraction of a second. If we were writing in bigger blocks (with write()), wait time would rise to a few seconds. That is the reason why your mp3 player occasionally skips if you have heavy writing in the background, its read request gets stucked behind many large write requests, and it sometimes takes many seconds before mp3 player gets its chance to access the disk.
Second picture is even more interesting. I started 4 seeker applications in parallel, which read randomly from the whole disk (/dev/hda) in 4K chunks. This is slightly similar to a server workload where you have mostly random & read disk operations. Disk is, of course, fully utilized (%busy = 100%). We see that average disk queue is around 4, because we have started 4 seekers. We also see that average request is around 4K in size (IIRC, on 2.4 kernel this would be 8K, because kernel does readahead on the block device level). There is effectively no merging taking place because access to disk is purely random. Disk service time is 13.5ms. If it was a SCSI 10krpm disk, it would be better. Also we can see that in this very demanding situation the disk is able to service around 72 requests per second. Good SCSI disks get over 100 request per second thanks to higher rpm and more robust construction. Not to mention good RAID10 arrays, they smoke on this test.
How fast is your disk?