Your assumption is correct. The kit I tested did not have any mechanical connections aside from the electrical mating with the baseboard through J1 and J2
I will caution you in applying "a lot of force" to the SOM. Althogh the connectors are rigid, the PCB will flex, thus applying to much force may possibly break one or more solder joints with the the connectos.
Do you have access to a microscope or magnifying glass? If you do, I would check the current condition of the solder connections between the SOM and connectors J1 and J2, especially the pins assocated with the JTAG.
-andrewa@logicpd.com