Media files are different from parts of your system that contain application logic in that serving media files is an I/O intensive task, whereas app logic usually requires some combination of I/O and CPU (the exact balance is very app-dependent). This is why it indeed makes sense to use a dedicated media-serving system that is optimized for disk and network throughput

Some general guidelines if you use your own dedicated server:
- Invest in lots of RAM and use caching for your most commonly-consumed content. The idea is to save on disk access-time (RAM is roughly 100 times faster in theory). Memcache is the most popular solution nowadays afaik

- Invest in fast disk IO, install multiple disks and use RAID (striping) to improve throughput

- When selecting a hosting provider for your dedicated / co-lo server(s), focus on bandwidth

- If possible, you want to locate the files close to their consumers in order to improve network latencies. So for example media files in Brazilian Portuguese would benefit if stored on a server in South America

- A good CDN can solve practically all of the above. In my own experience, it reduced the load on our own servers by ~85%. We use Cotendo and Akamai. Other providers you can look at ottomh: CDNetworks, Limelight, Level3

If you're just starting out then your best bet imo is using S3 to store your files, with CloudFront as your CDN. In my own experience, its a very simple solution to set up, and quite cost-effective
*when starting out* - as costslinearly with the amount of data and usage. Beyond a certain threshold though it makes sense to start looking at managing your own dedicated storage racks and use some other CDN.