Like all questions involving names and marriage, there is no set convention (at least in the US).  I know a male mathematician who publishes under his wife's last name which he took at marriage and I know people who have started publishing under a new name before they took it legally.  As Ben says, there's also no rules for names that don't involve marriage: not only do people pick whether they use their full first name or a nickname, some people use initials, and some people use nicknames which are not related to their legal first name.

Although there are no set rules or conventions, most people seem to agree that early on in your career it's unwise to change the name that you are publishing under.  Your name is your brand and diluting it is likely to hurt you professionally.  Thus there's a strong tendency for people to publish under a fixed name.  Nonetheless this is not a fixed rule, a particularly striking example is a *theorem* that's changed names: Nichols-Richmond nee Nichols-Zoeller.