Hey there!
Are you referring to a custom transform for Foundry Rules? You can find some documentation about it here.
If you follow these steps, you can deploy your own transform (in Java) that interacts with the Foundry Rules backend.
Please note that once you are using self-managed transforms, there will be manual maintenance effort. It will not update automatically with any workflow changes you might do within the Workflow Configuration after creating your custom transform.
Regarding limitations: The initial state of the self-managed transform should be as “capable” as the managed transform - there should be no special limitations.
What are you trying to achieve with the custom transform?
I have recently also posted about selective rule evaluation within Foundry Rules via a custom transform here.
Hope that helps!
Alex