The question is about the techniques I used to produce my collection of Dedekind eta-product identities. I have not kept detailed notes during the time I was working on the project, but the general ideas are simple. I found it was convenient to use PARI/GP for the calculations, but I could have used SageMath or Mathematica instead. You wrote:
These results were obtained through computer-assisted exploration and pattern-finding, rather than traditional mathematical proof techniques.
In fact, proof techniques are great for writing down proofs, but they are not of any use in finding the proofs or even the results to be proved. That requires human intuition, insight, and pattern recognition based on experimentation especially with computer assistance.
I would like to know is there any theory behind these computer programs
There certainly is some theory behind the computer program algorithms to help ensure that the calculations are mathematically valid. For examples of this you can refer to Donald Knuth's "The Art of Computer Programming". Sometimes, for finding solutions of Diophantine equations, the computer is instructed to perform systematic search with no assurance that all of the valid solutions will be found. Even here, there are techniques that can reduce the space of solutions to be searched. One example of this is the PARI/GP function hyperellratpoints(). All of the source code for PARI/GP is available under the GNU General Public License including for this function written based on Michael Stoll's 'ratpoints'.
As very relevant example is the results of Ralf Hemmecke for the construction of all polynomial relations among Dedekind Eta functions of a given level using an algorithm based on Groebner basis. In my own work for finding polynomial relations, I wrote a general purpose PARI/GP function to generate all monomials of a given degree for some list of power series and then it solved a system of linear equations based on the coefficients of the series expansions. This was sufficient for my purposes.
Any suggestions to improve skills in Experimental Mathematics especially in Ramanujan's work area.
I suggest that you familiarize yourself with some of Ramanujan's results. For example, I started reading from the beginning of his Lost Notebook and checked his equations by expanding them in power series. Then I tried to look for the coefficient sequences in the OEIS. This led to my question Ramanujan's Lost Notebook page 1 first equation and OEIS sequence A260195. Anther suggestion is to read my brief essay "A Multisection of q-series". You will probably find Bruce Berndt's five volume editing of Ramanujan's Notebooks helpful for studying the many amazing results found by Ramanujan. Same thing with George Andrews and Bruce Berndt's five volume editing of the Lost Notebook.