Add the line below to a file called /etc/modprobe.d/options. If the file doesn't exist, create it.
options snd-hda-intel model=3stack-6chNote that the model identifier you use will depend on the particular audio chipset installed on your system. The above is suitable for the Realtek ALC662 without S/PDIF - check the list here to find the one you need. Also, if you are using pulseaudio, e.g you're on Ubuntu, put the above line in /etc/modprobe.d/alsa.conf instead.
Reboot your system to make this change take effect. You should now be able to configure all 5.1 channels using alsamixer. Run this from the command line, and use the arrow keys to select the right-most option, Channel Mode. By default this is set to 2ch (ie stereo); press the up cursor key to change it to 6ch (ie 5.1). Make sure the PCM, Front, Surround and Center levels (or equivalent) are all at 100 (or above zero at least), and press Esc to quit alsamixer. You should now have 5.1 enabled.
If you don't see Channel Mode in alsamixer, it might be because your terminal window is not wide enough to show all audio options. Either make your terminal window wider (no need to shut alsamixer) or just keep pressing right arrow until alsamixer cycles through non-visible options.
You can test it by running this command:
speaker-test -Dplug:surround51 -c6 -l1 -twavSometimes, you may find that alsamixer doesn't remember the settings and you have to change the Channel Mode to 6ch every time you boot up. In such cases, the following command should save the settings permanently:
sudo alsactl store