Circulating tumor cell (CTC)- and/or tumor-derived extracellular vesicle (tdEV) loads in the blood of metastatic castration-resistant prostate cancer (CRPC) patients are associated with worse overall survival and can be used as predictive markers of treatment response. In this study, we investigated the quantity/quality of CTCs and tdEVs in metastatic castration-naive prostate cancer (CNPC) and CRPC patients, and whether androgen deprivation therapy (ADT) affects CTCs and tdEVs. We included 104 CNPC patients before ADT initiation and 66 CRPC patients. Blood samples from 31/104 CNPC patients were obtained 6 months after ADT. CTCs and tdEVs were identified using ACCEPT software. Based on the morphology, CTCs of metastatic CNPC and CRPC patients were subdivided by manual reviewing into six subclasses. The numbers of CTCs and tdEVs were correlated in both CNPC and CRPC patients, and both CTCs (p = 0.013) and tdEVs (p = 0.005) were significantly lower in CNPC compared to CRPC patients. Qualitative differences in CTCs were observed: CTC clusters (p = 0.006) and heterogeneously CK expressing CTCs (p = 0.041) were significantly lower in CNPC patients. CTC/tdEV numbers declined 6 months after ADT. Our study showed that next to CTC-load, qualitative CTC analysis and tdEV-load may be useful in CNPC patients.