Dear Chandramohan,
I am curious to understand how wireshark detects RTP protocol using UDP
dissector
There is the procedure (see epan/dissectors/packet-udp.c:
decode_udp_ports()). It works for any UDP based protocol, just for RTP
are a few additional setting.
Procedure how to use wanted sub-dissector for UDP based traffic (I'm
referring RTP):
1) There is record that the traffic is part of existing conversation and
it is RTP protocol. Conversation can be recorded as sourceIP:sourcePort
or destinationIP:destinationPort and combination.
Conversations are created by other dissectors which see signalization
that says that the traffic is RTP. E.g. SIP/SDP, H.323, SKINNY and others.
2) Table with well known ports for protocols (e.g. 53 = DNS) is checked.
There is no record for RTP, but a user can use 'Decode As' and set a
port as RTP manually (table can be saved for later use).
3) Wireshark (UDP dissector) blindly tries every UDP based protocol
which allows heuristics. If its sub-dissector says "OK", it is decoded
as the protocol.
Step 2 causes that if RTP uses well know port (there are many in RTP
range), it is never recognized as RTP ad is incorrectly decoded as other
protocol.
Step 3 causes (because protocols are just "tried") that it can happen
that traffic is guessed incorrectly. So even it is RTP, other
sub-dissector was tried before and said OK incorrectly.
There are settings related to steps 2/3:
i) Preferences -> Protocols -> UDP -> Try heuristics sub-dissectors first
ii) Analyze -> Enabled Protocols -> RTP -> rtp_udp (BTW you can
enable/disable many other protocols there).
When i) is checked, steps 2 and 3 are swapped. So RTP is recognized even
uses well known port. But it increases false positives.
When ii) is unchecked, no heuristics is made for RTP. So RTP is then
recognized only when signaling is seen in advance (step 1) or 'Decode
as' is used (step 2).
Best regards,
Jirka Novak