Easy as pie!  In the meeting invite, just go to the Actions menu and uncheck
Request Responses.

That's all you need, but, doing this, keep in mind that you won't know who
accepted/declined/tenative...d the meeting (iirc).

At an old company I worked for, we had visiting speakers come in and they'd
uncheck the option just for the reason you're stating.