You can set formation subs under your coach schemes tab. But that only pulls from the base playbook your team has. All that means is you can't pick your coach's playbook to be a custom playbook.
I personally still sub mine manually by formation every single game. These coach formations subs can get funky if you lose a player, sign a new player from FA or have an injury. It's been a bit buggy since it was added to the game a few years ago. Like when someone is injured, under your coach formation subs, it might show a certain guy starting twice in the same formation leading to the game putting some random bum from your bench in there.
You can try that out but it's mostly worthless if you use a custom playbook on defense imo.