1) Public bus, run by the Oman National Transport Company I have only ever taken these between cities (like Dubai to Muscat, for example), but ONTC does run in Muscat, albeit with very spotty service and only at certain designated spots. Other than that, my experience (perhaps in ignorance) is that it is basically useless for getting to points around the city quickly and efficiently. If you have an hour (or two) to wait, then I think it is decent alternative.
2) Bus/Van These run on many of the large roads in Muscat, and are incredibly cheap. 200-300 baisa. You wait (overpasses are often the best spot), board, and pay. You bang the roof or get the driver's attention when you want to stop. The busses are often crowded (typically with South Asian expats), you are liable to be subject to some pretty awful music choices (usually of the shab3ab variety), and if speeds over 120 kilometers per hour are not your thing, then I recommend another means. Nevertheless, they will get you cheaply and efficiently to most destinations along the main road, including al-Hail, al-Athaibah, as-Seeb, al-Khuwair (not inside), Mutrah, and Ruwi.
3) Taxis The average taxi is orange and white with a light on the top. My experiences with taxi drivers in Muscat have generally been positive. Drivers are generally professional, courteous and helpful. It helps to speak some Arabic (and they generally get very curious if you do), although many speak passable English. If you share a taxi, the price will go down; but drivers typically want you (especially if you are a white expatriate) to "engage", that is, to rent the whole taxi out from your pickup to your destination. This generally means the price will go up, but for out of the way places (such as Sultan Qaboos University) it is the only way to get them to take you. If you are new, ask the price beforehand, if not, they may ask you to pay an outrageous price at the end of the ride. I have found it is better to guesstimate the price, give them what you think is fair, and if they think it is unfair, they will tell you.
Here are some typical (fair) prices, based on my own experience.
from al-Mabailah to al-Khuwair: 3 Omani riyals
from al-Khuwair (Safeer Continental) to Grand Mall: 1.5-2 riyals
short distances within a neighborhood: 300-500 baisa (1/2 riyal)
from al-Hail to Sultan Qaboos University: 1.5 riyals
Your experience may vary, but these are some generally agreed upon prices. As a rule of thumb, if you are staying in the neighborhood for the ride, you should not pay over 1 riyal. The exception is leaving malls. This is a class issue. The drivers assume that since you have enough disposable income to be shopping at the mall, you have enough to pay them quite generously. I have been charged upwards of 5 riyals to leave Grand Mall and go to my apartment in al-Khuwair. If you don't mind walking through some parking lots to the main road, you can get a much cheaper price.
And that, my friends, is transportation in Muscat. A subway or train system would be a welcome addition. I wonder if the government is planning to install one? A train running from Mubailah to Mutrah (or even from Sohar to Muscat), would be an amazing and welcome addition to public transport options in this beautiful city.