Correct, without the controller in port one the infinity will wait up to 15 seconds before the unit will turn on. It does this because its waiting for possable commands from the controller on how to boot, because as you know you can boot the PS2 into different modes by holding certain buttons.
Also to boot the flash upgrade if your MI is a homebrew you hold
RESET until you see the blue light on the eject button, not holf the eject button its self
Also if its haveing a hard time even after you do it the right way this could be because the crappy laser in your V4 simply cant read it. You may need to adjust your laser. This is of course assuming your install is correct.
EDIT: AALLSSOO!!! why in the hell are you trying to get it to boot a CDRW! V5 and latter console will boot CDRW but struggle at that in most cases. Using CDRW's in a V4 is just a waist. Use a good quality CD-R.