The protocol filter name checks for the existence of the protocol entry in the proto_tree data structure hierarchy.
This is done with proto_tree_add_item(), as in this example from packet-ip.c :
ti = proto_tree_add_item(tree, proto_ip, tvb, offset, hlen, ENC_NA)
Could it be that your "proto_ABC" was not added to the tree in this way?
Or, maybe the value that was assigned to your protocol via proto_register_protocol() changed during execution?
As a test, compile "dftest" and run dftest with your protocol name, and see what the "display filter virtual machine" opcodes will be. Maybe your protocol name is matching something else?
Also, you can run: "tshark -G protocols" to see if your protocol really is known.
Gilbert