It is possible. In fact, there are three different ways to book a ticket Dublin-Glasgow:
(a) via Belfast/Stranraer (quickest)
(b) via Dublin/Holyhead or DL/Holyhead (cheapest)
(c) via Rosslare/Fishguard (not for the faint-hearted, expect about 12 hours on a train or in fact trains...)
Unfortunately it seems as if the most reliable way of getting your hands on a ticket is via the CIE travel office on Abbey St (or by ringing them). They actually issue the tickets with handwritten coupons, which is quaint but embarrassing.
I've done Dublin to various UK points over the last year, never had a problem with tickets. They will sell the return leg too (although prices are on a one-way basis). IF you're in the UK and trying to buy a ticket, most reasonably well-staffed station offices can issue the ticket (there's a full section in the National Fares Manual over there dealing with ship/rail tickets of various types).
has lots of information on your options (although from a UK perspective, so London-Dublin times come before Dublin-London, don't get caught by that!). I contributed a little bit of this page, but the site admin is a serious mine of information on international rail travel (whether leisure or business).
Apparently you can also buy the sail/rail tickets at the port, if you're starting with a ferry (i.e. Dublin Port-Holyhead-London), but I've never tried it.
Finally remember that the pricing is by zone (on both sides), so sometimes if you have time you can really get a nice trip out of it (i.e. the one-way via Holyhead ranges between €30 and €39 during the winter, but that €39 would get you as far as Inverness or Aberdeen if you wanted it, as everything beyond a certain point is all in the one, huge zone.