There's a good reason most ultralights, even in the early days, used 3-5 times this much power: because 20-30 hp is the minimum needed to give reasonably safe takeoff and climb performance. You will not meet your goal of 160 lb dry if you provide enough wing area to lift that plus a pilot; even a fabric hang glider type ultralight, by the time it has a seat, landing gear, and engine mount, will be at or above this dry weight -- and even with your goal dry weight, add a pilot and five gallons of fuel, and you'll need a long takeoff roll to reach a safe takeoff speed.
You've got around 90 lb to spare under the ultralight limit (assuming you're in the USA): I'd recommend allowing half of that for overruns in the airframe, and putting the other half into a larger, more capable, likely more reliable power plant. Get something that's actually intended for ultralights -- you'll get more power per pound, which makes everything else better, and you'll decrease the likelihood of a failure at a critical time (like at low altitude after takeoff).