A Cryptography Primer: Secrets and Promises

It does a good job of establishing all the basic parts and adding on to get to higher-level concepts. I think it helps that, instead of focusing on a bunch of algorithms that can be looked up, the book uses simplified models to focus on the important ideas.