Jonathan Korman (a mathematician) and Wing-Yee Tong (an artist) coauthored The NSA and the Social Responsibility of Mathematicians in the December 2016 issue of The Mathematical Intelligencer. It argues that "mathematicians have a moral obligation to make sure the power of mathematics is developed and used responsibly and not against public interest", in contrast to the general lack of involvement as evidenced e.g. by the observation that

... in the period January 2014 to February 2015, only 33 individuals with affiliations to mathematics departments have signed the declaration Academics Against Mass Surveillance. In contrast, a petition in support of continued funding for the Institute for Mathematics and Its Applications accumulated more than 2000 signatures over the one month period January 16 to February 16, 2015.

This cites another article Declining Mathematics Funding at the DoD, by Allyn Jackson in the January 2000 issue of the Notices of the AMS, which also discusses the AMS referendum referred to in the other answers. Here are a few relevant paragraphs:

It was almost twelve years ago that the AMS membership passed a referendum concerning support for mathematics from the DoD agencies. Motion 2 of the referendum expressed concern about the “tendency to distribute this support through narrowly focused (mission-oriented) programs and to circumvent peer review procedures.” The motion warned that this tendency “may skew and ultimately injure mathematics in the United States,” and ended by saying, “Therefore those representing the AMS are requested to direct their efforts towards increasing the fraction of non-military funding for mathematics research, as well as towards increasing total research support.” The referendum also included motions about the Strategic Defense Initiative and other, less controversial, matters of funding policy. The referendum drew about 7,000 votes, and motion 2 passed by a wide margin, with about 74% in favor and about 19% against (there were some abstentions).

At AMS meetings and in the pages of the Notices, passions flowed so hot that it is surprising to see how cool they are today. In retrospect the referendum seems like a quaint reminder of a less practical, more idealistic time that has since passed. The referendum’s main effect seems to have been to alienate, at least temporarily, certain segments of the mathematical community from the AMS. Some mathematicians were deeply offended by the referendum; one was James Crowley, who at the time was the head of the mathematics program at AFOSR and is now the executive director of SIAM. Today AMS representatives do not seem constrained to focus their attention only on “increasing the fraction of non-military funding for mathematics research.” As one observer put it, the AMS referendum is “not on the radar screen” of anyone concerned with funding for mathematics or science.

Indeed, the AMS has been active on a variety of fronts to try to boost funding for science and mathematics by all federal agencies, including those in the DoD. The AMS Committee on Science Policy, chaired by Arthur Jaffe, has invited to its meetings a number of key people from the DoD, including Robert J. Trew, director of research in the office of the secretary of defense. The AMS is active in the CNSR (Coalition for National Security Research), a group of sixteen scientific societies, university groups, and industry representatives who are advocates for increased support for research by the DoD. Perhaps their voices are being heard: The fiscal year 2000 defense appropriations bill, signed into law by President Clinton in October 1999, contains a 7.3% increase for DoD research. Consistent advocacy for all of science and mathematics, bolstered by alliances with other scientific groups, is what is likely to improve the funding picture for mathematics at the DoD agencies. As Rankin put it, “It is in the interest of mathematics to have all agencies that fund science be healthy.”