Using the delete key instead of backspace should do it. On a PC, anyway.

If you’re on a Mac, and therefore don’t have a real delete key (the delete key on the Mac’s keyboard is functionally equivalent to a backspace key on a PC), you can do fn + delete to accomplish the desired effect.

StackExchange / http://How to delete MS Word table cell contents but not cells

Talk about timeless solutions. This answer was posted in 2018. I can confirm this still works as of 2022 with Microsoft Word for Mac (version 16.68).