Almost all previously released version of WordPress is available for download in this WordPress archive page, including the very old initial release of WordPress at version 071-gold, 072-beta-1 and RC1. Other previous archives include version 1.0-RC1 and 1.0-platinum, 1.0.1-RC1, 1.0.1-miles, 1.0.2-blakey, 1.0.2, 1.2-RC1, RC2, beta, delta, mingus, 1.2.1, 1.2.2, 1.5-strayhorn, 22.214.171.124, 126.96.36.199, 188.8.131.52, 1.5.1, 1.5.2, 2.0 2.0.1-RC1, 2.0.1, 2.0.4, 2.0.5-RC1, 2.0.5-beta-1, 2.0.5, 2.0.6-RC1, RC2, beta1, 2.0.6, 2.0.7-RC1, RC2 and 2.0.7. Also available is the latest 2.1 branch or trunk of WordPress, from 2.1 beta1, beta2, beta3, beta4, RC1, RC2 to the final release of 2.1, and continue to version 3, 4 and so on.
ZIP and tar.gz archive of old WordPress files are available with their md5 checksum. But please note that these archives, except for the latest version, are not safe to use in production server, as it may contain security bugs or holes.
If you want to know what had been changed in each version upgrade of WordPress, visit the changelogs.