Sure, I’ll just write it out as simply as I can for the benefit of anyone who reads this without any knowledge of it.
SELCAL is, in effect, just a little gadget that listens to the HF frequency you’ve tuned to. When it “hears” the tones associated with its fixed code (e.g. AB-CD which corresponds to these tones), it gives a chime in the cockpit to alert the crew. This allows the crew to have the volume turned all the way down, because the SELCAL chime replaces the need to listen out for their own callsign.
In order to use SELCAL, you must check it first. This is done when first checking in with the oceanic controller as you enter the oceanic airspace (or when they log on), and each time you’re transferred to a new sector/controller. All you do is request a SELCAL check, and once you receive a positive chime, you acknowledge it as received and you can now turn the radio volume down. If ATC needs to talk to you, they will use SELCAL to alert you and you quickly turn the volume up and answer.
It’s worth keeping in mind that SELCAL only monitors the selected frequency, therefore you must stay tuned to the frequency!
I use this every time I fly through HF radio sectors, and it works every time. I make sure the same SELCAL code is shown in the FP and in the vPilot login screen. Once checked in, I ask for the check, and then I use the Windows volume mixer to adjust vPilot down to zero volume. I use a USB headset, with voice via headset and all other audio through speakers, which allows me to only turn down the radio volume. For headsets with audio jacks, I don’t know 100% whether you can turn down only the radio or if this means all audio from vPilot will be turned down, so beware that you might only get the visual warnings of the vPilot application if this is the case. For Fenix users, the ACP volume knob adjusts the vPilot levels in Windows volume mixer just as if you were to do it manually, so it works the same.
Edit to add: It can happen that multiple aircraft have the same SELCAL code, in which case all of them will be alerted. This can also occur IRL, because (at least until the recent update/expansion of the system which we don’t simulate yet) there are not enough SELCAL codes for all the aircraft around the world. The codes will therefore traditionally be assigned regionally based on where the individual aircraft usually flies as to minimise overlap.
I guestimate that on VATSIM the chance is higher for duplicate use, because the same addon+livery combo will placard the same SELCAL code in the cockpit, which increases the chances that two aircraft have the same code. Personally, I just pick some random letters that conforms with the rules for their combination as per a post further up, in order to avoid this issue (but then I have to remember it…). Of course, it’s perfectly legal to have overlapping codes in use with the same sector, but it will likely cause some confusion, so it’s nice to be aware of it.