First, this question is extremely broad and thus I hesitate to start answering it, but anyway some remarks.

Number therory is a abroad field and there are many different types of problems to which "computers" can contribute in one form or another.

Since you ask about rings of integers let me focus on this.

Regarding literature: One book I can recommend is Henri Cohen "A Course in Computational Algebraic Number Theory" and there is also a follow-up "Advanced Topics in Computational Number Theory".

In this book the author explains, among others, how to solve the basic tasks of Comptuational Algebraic Number Theory. So how to calculate with algebraic numbers, calculating rings of integers, discriminant, Galois group and so on. Also, certain methods of factorisation are discussed as well as question on arithemtic of polynomials.

The author is/was a main contributor to the development of Pari.
Pari is specialized for number theory; opposed to the other programms you mention.

Roughly, the functionality of Matlab is not geared towards number theory. Mathematics and Maple offer more here, certainly useful for some things and for some even very good as far as I know, but not specialized for number theory. An important (non-free) other program is Magma, which is I think considered as leading for certain number theory (related) tasks.

And, last but certainly not least, there is a large free open-source project Sage http://www.sagemath.org that has a certain focus on number theory (the founder William Stein is a number theorist). It inculdes (more or less) Pari and much other free open-source math software; some directly or indirectly relevant for number theory.

If you search for a possibility to do computational number theory and to potentially do something of lasting value, I would recommend that you look into Sage. Its web page offers a lot of documentation but also (number theory) papers written with the help of Sage. Yet also (number theory) lecture notes and text books with a computational slant. The developpment process seems very open and there are plenty of tasks to be done (from small to large, from beginner friendly to research level). [Note: I did not contribute anything to Sage, I only followed its developpment from a distance but somewhat in detail.]

First, this question is extremely broad and thus I hesitate to start answering it, but anyway some remarks.

Number therory is a broad field and there are many different types of problems to which "computers" can contribute in one form or another.

Since you ask about rings of integers let me focus on this.

Regarding literature: One book I can recommend is Henri Cohen "A Course in Computational Algebraic Number Theory" and there is also a follow-up "Advanced Topics in Computational Number Theory".

In this book the author explains, among others, how to solve the basic tasks of Comptuational Algebraic Number Theory. So how to calculate with algebraic numbers, calculating rings of integers, discriminant, Galois group and so on. Also, certain methods of factorisation are discussed as well as question on arithemtic of polynomials.

The author is/was a main contributor to the development of Pari.
Pari is specialized for number theory; opposed to the other programms you mention.

Roughly, the functionality of Matlab is not geared towards number theory. Mathematica and Maple offer more here, certainly useful for some things and for some even very good as far as I know, but not specialized for number theory. An important (non-free) other program is Magma, which is I think considered as leading for certain number theory (related) tasks.

And, last but certainly not least, there is a large free open-source project Sage http://www.sagemath.org that has a certain focus on number theory (the founder William Stein is a number theorist). It inculdes (more or less) Pari and much other free open-source math software; some directly or indirectly relevant for number theory.

If you search for a possibility to do computational number theory and to potentially do something of lasting value, I would recommend that you look into Sage. Its web page offers a lot of documentation but also (number theory) papers written with the help of Sage. Yet also (number theory) lecture notes and text books with a computational slant. The developpment process seems very open and there are plenty of tasks to be done (from small to large, from beginner friendly to research level). [Note: I did not contribute anything to Sage, I only followed its developpment from a distance but somewhat in detail.]