Sharing some feedback for Pilot on Dev Tier
I was able to rebuild an app I’d previously built in Developer Console w/ VSCode and Continue with three prompts and 25 minutes, pretty impressive
The one thing having reviewed the log files for the build is that at the 130k tpm rate limit in place for this tool on Dev(I assume it’s higher on Enterprise) >80% of calls failed on my first build.
It’s totally understandable why limits exist for us free users, however the limits are quite impactful, the build time would be significantly faster with a 225k tpm rate limit.
| Agent | Successful | Failed (Rate Limited) | Total Attempts | Failure Rate | Avg Tokens/Call | Peak Tokens/Call |
|---|---|---|---|---|---|---|
| ontology-architect | 3 | 14 | 17 | 82.4% | 27,582 | 35,522 |
| designer | 1 | 4 | 5 | 80.0% | 3,234 | 3,234 |
| frontend-architect | 17 | 71 | 88 | 80.7% | 38,133 | 52,662 |
| seed-data-generator | 1 | 5 | 6 | 83.3% | 26,304 | 26,304 |
| main-orchestrator | 4 | 16 | 20 | 80.0% | 25,285 | 29,064 |
| TOTAL | 26 | 110 | 136 | 80.9% |
Nonetheless, a slick tool.