Abstract: Highwall mining is one of the effective approaches for the safe and efficient recovery of coal resources trapped beneath highwalls in open-pit coal mines. In essence, it represents a boundary-dominated load-bearing problem primarily governed by slope geometry gradients and free-face effects. Existing coal pillar design methods based on tributary area theory or strength safety factors have difficulty in capturing the coupled response and abrupt instability mechanisms of the roof–coal pillar system. To address these limitations, a mechanical model of the roof-pillars system for highwall mining is established based on elastic foundation beam theory, incorporating slope-induced gradient loading. On the basis of characterizing the post-peak stiffness degradation behavior of coal pillars, a local mine stiffness analysis framework is introduced, and a dual-criterion stability criterion integrating both strength capacity and system constraint characteristics is proposed. A three-dimensional numerical model considering slope geometry and the layout of multiple entries is developed using FLAC^3D to systematically investigate the characteristics of the initial stress field in highwall support pillars and the spatiotemporal evolution of stresses during excavation. The spatial heterogeneity of the load-bearing behavior of the roof-pillars system along the entry depth direction is revealed. Furthermore, by integrating the strength-stiffness dual-criterion stability framework with load-bearing control mechanisms, a method for identifying critical zones in highwall mining is proposed. Taking the highwall mining project at the Antaibao open-pit coal mine in Pingshuo as the engineering background, theoretical analysis and numerical simulation are employed to effectively identify critical zones within the mining panel, based on which a preliminary coal pillar design scheme is determined. To address the challenge that personnel cannot access the entries after highwall mining, resulting in uncertainty regarding post-mining entry stability, a tracked robotic platform equipped with three-dimensional laser scanning and ultrasonic ranging sensors is deployed to measure the deformation of surrounding rock within the entries. The results indicate that the initial loads acting on highwall coal pillars under slope topographic control do not increase monotonically with slope height; instead, a transitional zone characterized by stress unloading followed by reloading exists in the lower portion of the slope. After mining is completed, the stress distribution of coal pillars along the entry depth exhibits a “decrease, increase, decrease” pattern, with the stress level and increment of central pillars exceeding those of boundary-adjacent pillars. Comparison between tributary area theory estimates and numerical simulation results shows that the tributary area method underestimates stresses in shallow-depth pillars while overestimating stresses in deep-depth pillars. Significant differences are observed in the load-bearing behavior of the roof-pillars system along the entry depth under varying boundary and loading control conditions. Based on dominant effect differences and stress response characteristics, the system is divided into a topographic effect-dominated zone, a slope gradient effect-dominated zone, and a three-dimensional spatial effect-dominated zone. Engineering case analysis demonstrates that the entry vicinity in the central part of the mining panel constitutes a stiffness-controlled critical zone, whereas the area near the vertical line of slope crest line at greater entry depths in the central panel represents a strength-controlled critical zone. Accordingly, coal pillar design dimensions in highwall mining should prioritize controlling both the strength and stiffness of these critical zones. Based on the preliminary coal pillar design scheme, in-situ testing conducted using the tracked robotic monitoring system confirms that the surrounding rock load-bearing system within the study area remains stable. Neither localized damage associated with abrupt instability in stiffness-controlled critical zones nor large deformations indicative of insufficient load-bearing reserve in strength-controlled critical zones are observed. These results verify that the coal pillar design method based on the strength-stiffness dual-criterion and critical zone control framework can effectively ensure the stability of surrounding rock in highwall entries and satisfy the requirements for safe highwall mining operations.