The place that SXMSammy is referring to is called La Cantina. They'll have a lunch special every day, on a chalkboard outside. The jewelry store that he mentioned is the Gustavia branch of Goldfinger.
The place that January referred to, Chez Ginette, burned down a year or so ago.
Right across the street from the ferry terminal is Le Repaire. It's not the least expensive place in town, but it is consistently good. You can often find some interesting specials on their chalkboard.
The least expensive place to eat in Gustavia is Cheeseburger in Paradise at Le Select. A cheeseburger and fries will run you about $10, and a Carib or Heineken will cost another $2 or $3, so $25-$30 for two.
A block uphill from Le Select is La Crêperie, a wallet-friendly place that specializes in crêpes.
On the far side of the harbor are two restaurants that get a lot of business from locals. One, on the harbor, is Côté Port. If you like goat cheese then try the Salade Pilou, a goat cheese salad with grilled shrimp and warm bacon dressing. Order the "Demi" (half) unless you have a huge appetite. A few hundred feet down the street on the inland side of the street is Carpe Diem. It's new this season (formerly Ti Zouk K'fé), and is becoming a very popular lunch and dinner spot with a menu that features pasta and grilled items.
You can find several handy St Barth maps at
https://www.sbhonline.com/st-barts-maps/You can find menus for many restaurants at
http://restaurants.alacarte-stbarth.comYou say that you don't want to rent a car, but depending on what you want to do, it may be less expensive than taking taxis. Rental cars can be delivered to the ferry terminal. It's been a long time since I've taken an island tour by taxi, but it's not going to be cheap.
On foot, you'll find that the roads out of town tend to be steep, narrow, and have no sidewalks. You might want to consider limiting your activities to Gustavia, and to Shell Beach on the edge of town.