Our experience with the removal of the factory front sights has shown that the longer they've been on, the more stubborn they are during removal. So, for the 580 & 581 series Mini's, they will typically come off with a little less effort.
Our instructions include a process that is intended to avoid marring the barrel. Patience is key to the process. The sight will come off. After removing the roll pin, there is nothing but tension holding the sight in place, so using a piece of hardwood to drive the sight forward is one effective way to do the job.
I usually put some masking tape around the barrel just behind the sight to protect the barrel's finish, and then drive off the sight using a scrap piece of aluminum stock and a hammer. Using some penetrating oil in advance is a good idea.