During the design phase, which is better, identifying the security mechanisms and techniques that will be used to protect the system (such as selecting the suitable encryption algorithm) before creating the threat modeling or during creating threat modeling.
I just want to know in which step in the design phase we should identify the security mechanisms and techniques that will be used to protect the system
It's not a question of which is better, to a large extent these activities come together.
For example, some of the trivial threats (e.g. SQL injection and XSS) can be properly mitigated upfront, by choosing appropriate design, using suitable technology, and defining relevant coding guidelines.
On the other hand, most security mechanisms and mitigations are irrelevant - until you understand the threats that you are trying to mitigate.
And yet another point to consider is that any security mechanism you implement, also needs to be threat modeled. Yes, sometimes a "security mechanism" actually introduces NEW threats, and this needs to be considered carefully. (E.g. anti-virus...)
So, bottom line: an iterative approach is best, involving the threat modeling as part of building the design.
Just as you do the system design iteratively - first the overall architecture, then detailed design, then specific modules/features/whatever - for each iteration there should be a relevant TM. So e.g. post-architecture/pre-design you can build a trivial TM; as you get into more detail, TM those details as you go!
I agree with what AviD says, and would go a step further.
Think of threat modeling as a set of activities, and different threat modeling activities happen at different times. Going breadth first during design helps you find threats overall, and going depth-first during implementation helps you find threats against the defenses you're building (this can be thought of as designinig test cases.)
If you're visual, this triangular interplay between threats, mitigations and requirements may be helpful. (https://adam.shostack.org/blog/2017/05/threat-modeling-iot/ (Updated to address broken link concern) )
So you'll iterate back and forth between 'here's a design; that allows these problems, so we change the design, and then we consider what can go wrong.'
Yes, sometimes a "security mechanism" actually introduces NEW threats, and this needs to be considered carefully. (E.g. anti-virus...)? – Motivated Feb 11 '16 at 07:04Why do you say that there is more risk with the installation of an antivirus?Because an antivirus is software. Software can have bugs. Bugs can enable threats.If an antivirus is deemed to be a reliable security mechanism, what is?Well, not getting in to whether or not its good (I don't like it for most issues, but it has value). Point is even a GOOD security mechanism is still software.How would you model security mechanisms?Same as any other software. Of course, in addition to whether or not it actually mitigates the intended threat, but also if it introduces any new ones. – AviD Feb 11 '16 at 16:49