I think I know the issue.

You can not change the grid from e.g. 4x2 to 4x1 if just one cell (even if hidden) spans over two rows in this case.

You need to make sure that any content only occupies one row to be able to reduce the grid to one row. In this example I try to reduce to one row; generally speaking your content should not spill to cells that would be affected by reducing your grid.

Might have been obvious to others; for me it was surprising. Maybe a better way to handle would be to (force) place the content in available cells if they span multiple cells when rows/columns are reduced.