Why didn't Hitler invade Spain and Portugal?

I was thinking about this the other day. If Hitler would have invaded Spain early on in the war, he could have severely cut off Allied effort in N. Africa by controlling Gibraltar. Portugal would have been useful as a port to the Atlantic, the Azores could be used as a strategic base. Spain and Portugal were both militarily weak during these periods with Spain losing many soldiers in their civil war. I don't imagine they would put up much of a fight against the Nazi war machine. Would have been walk in the park for the Germans, like France or the Low Countries. I understand Franco and Salazar were ideologically similar to Adolf Hitler, but Hitler demonstrated in the past that he had no qualms about betraying allies.