OK, the rubble can be removed with remove furniture or dug through by ordering your imps to dig. This will leave behind stones, that you can’t build on. If you really want to build on the same space again, you can actually claim the tiles where the stones are and “remove furniture” on them.
Thanks to PolarBear and ShadowLancer for this information.