A validation formula for a custom Outlook form must return True if the data is "good" and False if it is not. I would probably try to handle this validation with code in the Iem_Send or Item_Write event handler. I am averse to complex validation formulas (or lots of them).
But if you want to give it a try, think of each checkbox field individually. Your data is "good" if the checkbox is not visible or if the checkbox field value is True. When is the checkbox not visible? When the option button field has a value other than the other corresponding to the checkbox. So, the formula for each checkbox would be something like this:
( [CheckBoxField1] = True) Or ([OptionButtonField] <> 1)
where I picked 1 out of the air as the value that the OptionButtonField has that makes the check box visible.

Signature
Sue Mosher, Outlook MVP
Author of Configuring Microsoft Outlook 2003
http://www.turtleflock.com/olconfig/index.htm
and Microsoft Outlook Programming - Jumpstart for
Administrators, Power Users, and Developers
http://www.outlookcode.com/jumpstart.aspx
> OL custom form
>
[quoted text clipped - 10 lines]
>> >
>> > Is this possible? It's a little complex for my current skills