There are two clues here. The first is "database" - Access is the database
application.

The second is that, in spite of that, you fancy Excel, probably because you
are more comfortable with Excel.

It is your call.  The probability is that sometime someone somewhere,
perhaps even you, will ask "Why Excel?", and perhaps not so politely.

My guess is that Excel will do what you require now, but businesses change
and hopefully grow, and it will be when the business is growing fastest and
everybody is at their busiest that the limitations become obvious. Your
call.

What question next? Shall we use Access or Excel for Inventory?  For orders
?.....

You already have the answer there! There will be many visits to each
customer, which constitutes a typical one-to-many relationship in
database terms... definitely a database application. I'm not saying you
can't do it with Excel, but then again I'm not saying you can't run
coast to coast either (Forrest Gump did it a few times!).
Furthermore, when you get started you'll come up with another fifty
things it would be nice to do with your app, and that will be fairly
easy with Access, but increasingly hard with Excel.
Of course, if you're not familiar with Access (which I take it is the
case, or you would have already started without wondering), it will take
some time until you get it to work, as Access's learning curve is quite
steep. Yet, if you can afford the time and effort, it will definitely
pay back. On the upside, Access comes with a few templates for typical
applications, one of which is Contact Management, which you may find is
90% or more of what you need! Check it out.

HTH,
Nikos