In the present work flame spray pyrolysis synthesis and characterization of the nano-scale phosphors; M'-YTaO4 and M'-Y(Ta0.85Nb0.15)O-4 have been studied for the first time. Phase and elemental analysis of the produced nanophosphors were carried out by X-Ray Diffraction (XRD) and X-Ray Photoelectron Spectroscopy (XPS) measurements, respectively. The surface morphology and particle size of the nano phosphors were identified using scanning electron microscopy (SEM). The XRD results reveal that the nanophosphors have monoclinic M'-YTaO4 and M'-Y(Ta0.85Nb0.15)O-4 phases belonging to the presence of M'-form of fergusonite structure. The particle sizes of the nanophosphors were found to be in the range of 50-100 nm. The spectroscopic characterization was performed by both radioluminescence (RL) and thermally stimulated luminescence (TSL) measurements after exposure to X-ray irradiation. Also, photoluminescence and decay times were investigated under UV excitation. The nanophosphors can be concluded as appropriate emissive materials for imaging, display and scintillator applications due to the efficient photoluminescence, moderate radioluminescence (RL) and thermally stimulated luminescence characteristics. (C) 2017 Elsevier B.V. All rights reserved.