Modifier Key when dragging to NOT smart update layer position

I wish I could hold down something like control, option, command, and/or shift when dragging to KEEP the dragged layers position in the layer list. This would be a huge benefit!