If you and/or your spouse live in a Schengen country, the situation is as follows:
As holder of a residence permit issued by a Schengen country, you need no visa for short-term stays in other Schengen countries - so you need no visa for Germany either. In any 180 days period you may stay for up to 90 days maximum in another Schengen country. Please check with the relevant authorities in your country of residence whether your spouse's residence permit entitles him or her to visa-free travel within the Schengen area.
If you or your spouse live in an EU country where the Schengen Agreement does not apply or not apply in full, the situation is as follows:
If the authorities of your country of residence have issued your spouse with a "residence card of a family member of a Union citizen" pursuant to Art. 10(1) of Directive 2004/38/EC, he or she needs no visa for travel within the EU.
If you and your spouse live outside the EU, your spouse will need an entry visa if he or she is from a country subject to a visa regime. In this case, however, a facilitated visa procedure applies.
If you have further questions, please visit the website of the relevant German mission or contact the mission directly.
The EU provides a guide to your rights as an EU citizen.