Every bearing/retainer interface I've come across has been a snug push fit; i.e. no need to get the bearing pressed into the housing. It doesn't exactly 'drop in' but doesn't put up too much of a fight. It's a tight enough fit that pressure on the outer of the housing causes the bearing to spin, rather than the bearing to spin in the housing, if you know what I mean. This assumes that all parts are clean, rust free and lightly lubricated (not the outer part of the housing that contacts the belt, obviously) when assembled.
The only time I'd be concerned is if the bearing is a sloppy fit in the housing. If you buy the one-piece part of course this area of concern is obviated.