Over on YouTube the popular security and hacking themed channel Hak5 have created two videos together with Mike Ossmann (creator of the HackRF and Yardstick One) that give a good introduction and overview on reverse engineering unknown radio protocols. In the video they show how to use a SDR like the RTL-SDR or HackRF to initially capture the radio signal, and then how to use the Yardstick One to reverse engineer and recreate the signal. Using this process they reverse engineer the radio protocol for a wireless liquor cabinet lock.
The Yardstick One is a computer controlled wireless transceiver (but it is not an SDR). The Yardstick One understands many radio protocols by default and can be programmed in Python, lowering the learning barrier for reverse engineering signals.
Mike Ossmann has also been slowly releasing very detailed video tutorials about DSP and radio related topics. If you are interested in reverse engineering radio signals it is a very helpful series to watch.