The most powerful FIR design method is Parks-McLellan, often called Remez or Remez Exchange. I make no claim to understand the maths. Some years ago I used a 30 day free trial version of the expensive Nuhertz filter design software to do some "brick wall" filters with many dozens of taps.

http://www.nuhertz.com/Somebody made the point that FIRs are linear phase. This is true if the coefficients are symmetric but not otherwise.

For conventional Butterworth, Bessel etc filters an IIR will usually implement in less hardware/processor cycles than a FIR. Of course it won't be linear phase.

Thanks for mentioning Octave - I hadn't realised thre was a free alternative to Matlab. Looking around, it seems there are some Remez tools available for Octave.