Creating personalized and animatable 3D avatars is challenging, with real-world applications in gaming, virtual try-on, animation, and VR/XR. On the other hand, it is also a complex problem to infer cloth geometry and dynamics from sparse and monocular view data. Existing methods for modeling 3D humans from depth data have limitations in terms of computational efficiency, mesh coherency, and flexibility in resolution and topology. Reconstructing shapes using implicit functions and extracting explicit meshes per frame is computationally expensive and cannot ensure coherent meshes across frames. Conversely, predicting per-vertex deformations on a pre-designed human template with a discrete surface lacks flexibility in resolution and topology. To overcome these limitations, we propose a novel method NSF: Neural Surface Fields' for modeling 3D clothed humans. A distinctive aspect of NSF is that it defines a neural field solely on the base surface, enabling it to predict a continuous displacement field over the surface. To determine the shape of the base surface, our method fuses depth observations in a canonical space and learns a coarse geometry without high-frequency pose-dependent deformations. Compared to existing approaches, our method eliminates the expensive per-frame surface extraction while maintaining mesh coherency, and is capable of reconstructing meshes with arbitrary resolution without retraining.