I really like this nixCraft Howto for Red Hat simple networking issues. After putting the new MAC address on the HWADDR line in /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-eth1 (just added 1 to eth0's MAC addr), I ran "/etc/init.d/network restart" expecting everything to come up. No such luck - got Fatal "Device eth0 does not seem to be present" errors. Verified the MAC address, swapped the addresses between eth0 and eth1, restarted the server, no joy.
Eventually someone smarter than me to me to look at eth2 and eth3. Turns out the NIC card was binding to eth3. To resolve this, I commented out the HWADDR line in ifcfg-eth0 and ifcfg-eth1 and restarted the server. Running "ifconfig eth0" showed the NIC now attached to eth0. I wanted this to always be the case so now I uncommented the HWADDR line in ifcfg-eth0 and ifcfg-eth1 and restarted again. I probably could have just restarted network not the server and got the same results but everything was good.