Depending on your state, the code of professional responsibility for attorneys may require that the unearned portion of a retainer, that is paid with the expectation that fees for service will be charged against the retainer, be refunded when the attorney-client relationship is severed. Google "[your state] failure to return retainer".
If you do obtain new counsel and the CPR in your state is as described above, you are well within your rights to ask for a full accounting of billed hours and return of the remainder. In CA (my state), attorneys have been disciplined and suspended by the Bar for failure to return unearned fees. (Back in the 80s, my husband saw more than one of his law school peers surrender their licenses in lieu of disbarment, for failure to return unearned fees and misappropriation of client funds held in trust. Cocaine is a helluva drug.)