One of the smartest things I've ever done was to install a woodstove in my house.
I live in an 80 year old house with a 50 year old oil furnace. When I first moved in, I had the tank filled. It was $650. I asked the delivery guy how long he thought the oil would last. When he nonchalantly estimated six weeks, I knew that wasn't going to be viable, and went out the next day, armed with cash from the sale of my previous house, to buy the stove. No regrets, going into my fifth heating season.
It is hands-down the least expensive way to provide heat, and it's off-grid, which is a wonderful thing. You don't have to worry about power failures — as long as you can get to the wood pile, you'll be warm. At times last year that was a challenge, as I had to dig through four feet of snow, but hey, it's good exercise, and got me outside to experience the power and beauty of nature.
Last year, in the eastern part of the US we had an amazing amount of snowfall. My area had a total of 101 inches. It was cold. I heated my house for $360. No, not for one month — the entire winter.
'Start-up costs' will vary. My stove was $2200. Installation more than doubled that, but I had to have the brick chimney lined and repaired. Some installations can be just a few hundred dollars; it just depends on where you want your stove to go.
If I was going to do this again, I would emphatically not choose a stove with a catalytic combustor. These are devices that re-burn the particulate matter in smoke as it exits the chimney, thus producing a cleaner burn. They are also more efficient, so you'll use less firewood. However, if you are new to heating with wood, you'll probably ruin your first two before you get the hang of things. They are very fragile, hard to find and expensive to replace (about $200). There isn't a lot of information about how to burn using a catalytic stove.
Best advice — never close the damper until all the moisture has burned out of the wood. Moisture is what "kills" them.
Fuel (wood) is available everywhere, sometimes for free. Stacking a good looking cord of firewood is one of the most satisfying outdoor activities ever on a crisp fall day.
Wood burning also contributes to the local economy, as you get your wood from the guy down the road who cut the trees, split the logs and delivered them, not a huge corporation. All of the guys I've bought from lead economically marginal lives, and I'm happy to directly support the cash economy. Win-win all around.