2D A few objects Ignoring Collision causing insane lag

I have some enemies that should not be colliding with each other, so I used the Ignore-collision matrix. However, when only around 5 enemies or so collide with each other while moving, it drops the frame rate below 10 fps. Some of the enemies are large, and collisions happen constantly, so this is a critical problem. The enemies do have dynamic rigid bodies, and move around and rotate to follow the player.