I believe there's another possible approach here:
Register the dissector once with proto_register_protocol (as usual), which assumed caller is TCP
register another dissection function (for SMP) using
create_dissector_handle_with_name called something like "smp.tds"
than look for this name when retrieving the dissector handle in the TDS dissector
This means you should have 2 'entry point' functions to your dissector (usually there's only one dissect_PROTO).
This way by writing different dissection/payload handling code in those two functions you can react differently to different calling protocols (tcp calls will trigger the first function, TDS will trigger the second etc)