Paying the bills, doing the taxes, going through the mail are tough jobs to split up, because they don't necessarily lend themselves to multiple hands. They all need to be rolled together, because being a good bookkeeper and office manager requires setting up a system that is tightly bound to the quirks and working habits of the pencil pusher.
If your partner isn't much help with the the bills and paperwork, it's not going to help to demand that he take over a huge portion of it or pick it up on odd-numbered days of the month. If you've been minding the store, you don't want to introduce friction into the system by confusing responsibilities. It will be frustrating for both of you. Show him how you like to work and see if you can identify ways to communicate more effectively. Or if he handles the books and you feel cut out, ask how you can make his job easier by saving the right receipts, forwarding e-mails or putting important dates in the right place for him to process them.
Can you cure your spouse of his bad habits?