Velvet blazers are by their nature both extravagant and luxurious, and should be worn as such. They are purely evening pieces in my view.
Outfitwise, a burgundy blazer would make a great alternative top-half for a black tie look, especially with black velvet slippers or an evening scarf. For a more casual look, it would work with a crisp, white button-down and slim jeans, maybe with some solid loafers.
In general, keep it for the evening and always make it the focus of any outfit you put together.