Do you have soldering equipment?
You only have to put the minus of the LED directly to the flashlight body.
So you can solder that (usually back, sometimes white) wire to the first 3 pins of the FET.
You will then use the driver board for the plus spring only.

The driver will still be “ON”, but it uses a negligible amount of energy.

You will not have any low voltage warning or protection anymore.
But you will notice dramatic loss of output when the battery is running low.
This will happen before the battery is below 3 Volts, so it should be okay.