I could not wait for tomorrow 
You can get xiao_ble_sense kinda working with sysbuild and without mcuboot using the factory uf2 loader if you add a pm_static.yml similar to this:
adafruit_boot:
address: 0xf4000
end_address: 0x100000
region: flash_primary
size: 0xc000
app:
address: 0x27000
end_address: 0xec000
region: flash_primary
size: 0xc5000
sd_partition:
address: 0x0
end_address: 0x27000
region: flash_primary
size: 0x27000
storage:
address: 0xec000
end_address: 0xf4000
region: flash_primary
size: 0x8000
sram_primary:
address: 0x20000000
end_address: 0x20040000
region: sram_primary
size: 0x40000
This will instruct sysbuild to know where to place the application (0x27000, just where the adafruit bootloeader will jump to).
The issue is that it wont generate a uf2. But it will generate a merged.hex (even without mcuboot) that you can convert to a uf2 and it will run properly.