Create a shell script /usr/local/bin/magnet containing:

#!/bin/sh
/usr/bin/transmission-remote --add "$1"

chmod +x /usr/local/bin/magnet

In Firefox go to about:config

Right-click on any link and select "New->Boolean".

Boolean name: network.protocol-handler.expose.magnet
Boolean value: false

Right-click on any link and select "New->Boolean"

Boolean name: network.protocol-handler.handler.external.magnet
Boolean value: true

Restart Firefox

Next time you click a link of protocol-type magnet you will be asked which application to open it with. Point it to /usr/local/bin/magnet.