What would be the purpose of limiting to HID drivers??
As far as I read, bad usb will reprogramm the memory controller to act as a HID besides its memory controller functions.
The security risk then comes from hidden keyboard inputs. So the goal would be to limit the stick to storage devices.
My naive idea is to check if there is one and JUST ONE device connected to the host and if this device is a storage device, eveything should be fine.