I am new to mainframe security testing practice. So far I have read a few articles on that even security of mainframe can be compromised.
I would like to know if there are any such checklists that the security tester follow while testing a mainframe application.
Anything can be compromised, however traditional mainframe configurations are very well structured so generally it is considered much harder than, for example, an average server running PHP:-)
Start with one of the security hardening guides for your mainframe. If you are running IBM, the manuals on IBM mainframe security will give you 95% of what you need.Start with the redbooks. The one I have linked to is over 500 pages and it is just one of many!
That said - if you are new to mainframe security, you really shouldn't be going anywhere near a mainframe. Generally they run high value data and tools...
I think you'll have fun getting to grips with Mainframe security, it's quite a different world.
As @roryalsop says be very careful before you test a live mainframe, they tend to run very high value apps and can be fragile (i.e. if you break it, the test will likely end pretty quickly)
Some resources I would recommend
