The value behind is_overridable_library can be set through the method property_overridable_library_set() of an object. The following example demonstrates how to set a custom property named "prop" to be overridable.
import bpy
obj = bpy.context.object
obj.property_overridable_library_set('["prop"]', True)
The use_soft_limits is dynamically evaluated and it's not necessary to set it when using the Python API. If the soft_max or soft_min are different than the max or min, it will automatically show as enabled.
Calling bpy.ops.wm.properties_edit() results in an error message, because it expects that self._last_prop is set. This only happens in the invoke() function, not when calling execute() directly. Therefore the execution context has to be set by passing INVOKE_DEFAULT.
bpy.ops.wm.properties_edit("INVOKE_DEFAULT", data_path="object", property="prop", value="1.0", default="1.0", min=0, max=1, use_soft_limits=False, is_overridable_library=True, soft_min=0, soft_max=1, description="", subtype="NONE")
The downside of this is that the UI popup appears and has to be confirmed. Therefore the operator cannot be used without user interaction.
_RNA_UI, I'd avoid using a private variable from an undocumented part of the API and instead update it throughbpy.ops.wm.properties_edit(). – Robert Gützkow Mar 08 '20 at 11:52is_overridable_librarythrough_RNA_UIdoesn't seem to work either, when modifying an existing property. I'll have to look at the source code as see what's going on. Sorry for causing any confusion. – Robert Gützkow Mar 08 '20 at 12:38