This chapter is devoted to laying the algebraic foundations for border bases of ideals. Using an order ideal
O\mathcal{O}, we describe a zero-dimensional ideal from the outside. The first and higher borders of
O\mathcal{O} can be used to measure the distance of a term from
O\mathcal{O} and to define
O\mathcal{O}-border bases. We study their existence and uniqueness, their relation to Gröbner bases, and their characterization in terms of commuting matrices. Finally, we use border bases to solve a problem coming from statistics.