Topologists' answer: often in topology it is useful to study the fundamental group of a surface (e.g.) with boundary with a basepoint sitting on a boundary component. In algebraic geometry, we don't really have surfaces with discs sliced out of them, only surfaces with punctures. But it turns out that a tangential basepoint at the puncture is a good substitute which makes sense algebraically. You have a circle's worth of choice of tangential basepoints at the puncture, which matches up exactly with the circle's worth of choice of basepoints on the boundary component (which is a circle.)
Number theorists' answer: Grothendieck's section conjecture asserts that, when X is a proper smooth variety over Spec k, for k a global field, the conjugacy classes of sections from Spec k Gal(k) (a.k.a. pi_1^et(Spec k)) up to pi_1^et(X) are in bijection with X(k). When X is not proper, this isn't true -- but (conjecturally) the sections are in bijection with the union of X(k) and the tangential basepoints.
Ultimate answer: if Deligne's monograph on the fundamental group of P^1 minus three points is not one of the papers where you've seen this, read that. (But this is a big, long paper and the tangential basepoints are near the end, so don't read unless you also want to learn about motives, periods, etale fundamental groups, Malcev completion, etc etc.)
