The influence of the charge, structure, and host–guest interactions of different soft polymeric components on their assembly with negatively charged inorganic gold and up-converting nanoparticles (NPs) functionalized with cyclodextrins (CD-AuNPs and CD-UCNPs) was studied. The inter-/intramolecular interplay was evaluated by changing the morphology and the charge of the polymeric component. Use of a linear, anionic guest-functionalized polymer resulted in the formation of thin coating layers around the CD-AuNPs and CD-UCNPs without further NP aggregation. When the branched cationic polymer poly(ethylene imine) (PEI) was used instead, small clusters were obtained before the electrostatic neutralization point, whereas large NP aggregates were obtained above the neutralization point. When branched cationic guest-functionalized dendrimers were used, clusters were obtained at a 1:1 host/guest stoichiometry. In this case, self-assembly occurred when still far away from the neutrality point, indicating that this cluster formation is driven by supramolecular host–guest interactions. The luminescence of the CD-UCNPs with the linear polymer was quenched due to C–H vibrational modes of the guest polymer, whereas in the case of electrostatic cluster formation by PEI the luminescence of CD-UCNPs was enhanced after the neutralization point, due to the formation of densely packed NP assemblies. These findings should be useful for the design of hybrid polymeric–inorganic assemblies for sensing and biomedical applications.