Now that I see your photo I got the protrusion estimate wrong!
It should go in a bit more than that but not much more.
I'm not sure what that tool is you're using, but I use a flat-blade screwdriver
to wind in the piston.
As Peter said, make absolutely sure that the handbrake lever is fully back, and that
it moves freely.
One other item inside the piston is a spring that wraps around the self-adjust screw.
This spring, with a tag on it, acts to prevent the self-adjust mechanism from
unwinding itself by gripping the screw in one direction and allowing free movement in the other.
The tag can break off causing the self-adjust to malfunction.
This may be causing your issue, but I can't recall exactly its
relationship to the winding in screw.
Nigel