Leaving the battery inside the flashlight should be safe and will probably not raise any concerns.