[root@ceph1 ~]# ceph health detail
HEALTH_ERR Reduced data availability: 3021 pgs inactive, 23 pgs down, 23 pgs stale; Degraded data redundancy: 3021 pgs unclean, 1879 pgs degraded, 1879 pgs undersized; 1 stuck requests are blocked > 4096 sec
PG_AVAILABILITY Reduced data availability: 3021 pgs inactive, 23 pgs down, 23 pgs stale
pg 4.fcd is stuck inactive for 436215.668918, current state activating+undersized+degraded+remapped, last acting [2147483647,35,116,96,157,83,54,22,124,73]
pg 4.fce is stuck inactive for 411702.406372, current state activating+undersized+degraded+remapped, last acting [2147483647,193,97,51,27,92,74,154,178,129]
pg 4.fcf is stuck inactive for 436214.628338, current state activating+remapped, last acting [169,13,125,82,29,103,164,152,114,60]
pg 4.fd0 is stuck inactive for 434071.553971, current state activating+undersized+degraded+remapped, last acting [130,123,84,2147483647,47,67,5,158,99,61]
pg 4.fd1 is stuck inactive for 183022.405365, current state unknown, last acting []
pg 4.fd2 is stuck inactive for 242172.271319, current state activating+remapped, last acting [177,155,191,24,78,187,126,60,178,47]
pg 4.fd3 is stuck inactive for 436215.250978, current state activating+undersized+degraded+remapped, last acting [2147483647,82,32,176,4,164,184,125,168,117]
pg 4.fd4 is stuck inactive for 51415.912644, current state activating+remapped, last acting [14,89,114,65,50,47,108,151,24,124]
pg 4.fd5 is stuck inactive for 183022.405365, current state unknown, last acting []
pg 4.fd6 is stuck inactive for 61189.362405, current state activating+undersized+degraded+remapped, last acting [20,2147483647,109,131,156,55,68,146,40,94]
pg 4.fd7 is stuck inactive for 266944.251347, current state activating+remapped, last acting [27,35,75,140,98,87,120,12,55,183]
pg 4.fd8 is stuck inactive for 50579.572105, current state activating+undersized+degraded+remapped, last acting [107,92,118,68,156,2147483647,54,139,136,22]
pg 4.fd9 is stuck inactive for 436214.643977, current state activating+remapped, last acting [41,182,167,14,143,71,175,120,92,22]
pg 4.fda is stuck inactive for 436214.649384, current state activating+undersized+degraded+remapped, last acting [131,21,82,162,78,94,60,112,156,2147483647]
pg 4.fdb is stuck inactive for 50057.244902, current state activating+remapped, last acting [33,81,176,66,40,124,120,57,149,9]
pg 4.fdc is stuck inactive for 436214.627903, current state activating+undersized+degraded+remapped, last acting [134,61,16,121,47,68,177,23,2147483647,85]
pg 4.fdd is stuck inactive for 50780.389500, current state activating+remapped, last acting [174,66,167,188,19,135,153,105,15,42]
pg 4.fde is stuck inactive for 434089.629488, current state activating+remapped, last acting [79,176,35,155,65,146,181,161,120,59]
pg 4.fdf is stuck inactive for 50230.318560, current state activating+undersized+degraded+remapped, last acting [105,119,49,7,23,171,79,42,2147483647,182]
pg 4.fe0 is stuck inactive for 50579.574414, current state activating+remapped, last acting [106,81,181,178,192,31,49,171,48,8]
pg 4.fe1 is stuck inactive for 183022.405365, current state unknown, last acting []
pg 4.fe2 is stuck inactive for 408052.210808, current state activating+undersized+degraded+remapped, last acting [121,5,167,187,177,33,169,2147483647,127,190]
pg 4.fe3 is stuck inactive for 50049.175161, current state activating+remapped, last acting [191,42,23,81,114,64,56,12,103,127]
pg 4.fe4 is stuck inactive for 436214.644546, current state activating+undersized+degraded+remapped, last acting [48,135,94,11,188,2147483647,33,171,80,60]
pg 4.fe5 is stuck inactive for 51402.335662, current state activating+undersized+degraded+remapped, last acting [9,67,2147483647,193,30,53,94,83,118,132]
pg 4.fe6 is stuck inactive for 51588.682603, current state activating+undersized+degraded+remapped, last acting [154,74,53,29,124,164,2147483647,178,105,85]
pg 4.fe7 is stuck inactive for 436215.374741, current state activating+undersized+degraded+remapped, last acting [2147483647,119,62,138,4,95,71,81,160,35]
pg 4.fe8 is stuck inactive for 266998.114694, current state activating+undersized+degraded+remapped, last acting [19,183,10,99,76,111,34,2147483647,90,166]
pg 4.fe9 is stuck inactive for 245161.558491, current state activating+undersized+degraded+remapped, last acting [160,71,39,128,109,49,156,91,106,2147483647]
pg 4.fea is stuck inactive for 61219.768321, current state activating+undersized+degraded+remapped, last acting [160,72,62,6,96,117,147,44,2147483647,135]
pg 4.feb is stuck inactive for 50240.523773, current state activating+undersized+degraded+remapped, last acting [138,169,89,2147483647,36,156,95,29,63,121]
pg 4.fec is stuck inactive for 433531.193888, current state activating+undersized+degraded+remapped, last acting [90,125,16,74,123,168,2147483647,45,186,20]
pg 4.fed is stuck inactive for 242130.516170, current state activating+undersized+degraded+remapped, last acting [5,171,22,189,180,34,186,2147483647,187,52]
pg 4.fee is stuck inactive for 50049.156886, current state activating+remapped, last acting [152,39,125,168,186,114,22,8,184,174]
pg 4.fef is stuck inactive for 242130.514804, current state activating+remapped, last acting [22,52,89,138,151,111,78,34,103,157]
pg 4.ff0 is stuck inactive for 245793.017534, current state activating+undersized+degraded+remapped, last acting [37,87,25,181,14,102,70,57,119,2147483647]
pg 4.ff1 is stuck inactive for 358433.719053, current state activating+undersized+degraded+remapped, last acting [31,63,165,122,157,172,125,74,175,2147483647]
pg 4.ff2 is stuck inactive for 407935.143022, current state activating+undersized+degraded+remapped, last acting [16,91,177,56,113,158,2147483647,65,38,135]
pg 4.ff3 is stuck inactive for 50978.616817, current state activating+undersized+degraded+remapped, last acting [74,100,57,188,80,132,44,28,2147483647,157]
pg 4.ff4 is stuck inactive for 436214.660459, current state activating+undersized+degraded+remapped, last acting [62,2147483647,42,120,106,92,28,155,75,181]
pg 4.ff5 is stuck inactive for 408007.762799, current state activating+remapped, last acting [89,148,119,156,163,106,32,171,130,53]
pg 4.ff6 is stuck inactive for 407968.811566, current state activating+undersized+degraded+remapped, last acting [7,56,66,178,190,94,2147483647,84,159,183]
pg 4.ff7 is stuck inactive for 356275.532916, current state activating+undersized+degraded+remapped, last acting [176,62,31,15,87,126,65,180,2147483647,47]
pg 4.ff8 is stuck inactive for 356272.730814, current state activating+undersized+degraded+remapped, last acting [173,185,34,21,184,14,120,2147483647,175,133]
pg 4.ff9 is stuck inactive for 436214.647166, current state activating+undersized+degraded+remapped, last acting [188,68,29,125,102,143,2147483647,45,156,55]
pg 4.ffa is stuck inactive for 433531.194585, current state activating+remapped, last acting [139,173,25,135,49,154,67,164,179,101]
pg 4.ffb is stuck inactive for 245793.850376, current state activating+remapped, last acting [83,47,182,113,59,11,103,27,184,147]
pg 4.ffc is stuck inactive for 245170.487841, current state activating+undersized+degraded+remapped, last acting [157,38,2147483647,172,121,161,104,73,61,130]
pg 4.ffd is stuck inactive for 356279.246180, current state activating+remapped, last acting [48,161,172,154,72,135,119,146,103,50]
pg 4.ffe is stuck inactive for 408018.419952, current state activating+remapped, last acting [156,175,33,169,135,85,165,55,148,178]
pg 4.fff is stuck inactive for 358419.254932, current state activating+undersized+degraded+remapped, last acting [44,12,185,125,69,29,119,102,81,2147483647]
PG_DEGRADED Degraded data redundancy: 3021 pgs unclean, 1879 pgs degraded, 1879 pgs undersized
pg 4.fcd is stuck unclean for 471269.651318, current state activating+undersized+degraded+remapped, last acting [2147483647,35,116,96,157,83,54,22,124,73]
pg 4.fce is stuck undersized for 9349.487372, current state activating+undersized+degraded+remapped, last acting [2147483647,193,97,51,27,92,74,154,178,129]
pg 4.fcf is stuck unclean for 471264.748945, current state activating+remapped, last acting [169,13,125,82,29,103,164,152,114,60]
pg 4.fd0 is stuck undersized for 50232.767224, current state activating+undersized+degraded+remapped, last acting [130,123,84,2147483647,47,67,5,158,99,61]
pg 4.fd1 is stuck unclean for 183022.405365, current state unknown, last acting []
pg 4.fd2 is stuck unclean for 471262.989978, current state activating+remapped, last acting [177,155,191,24,78,187,126,60,178,47]
pg 4.fd3 is stuck undersized for 50149.017676, current state activating+undersized+degraded+remapped, last acting [2147483647,82,32,176,4,164,184,125,168,117]
pg 4.fd4 is stuck unclean for 471267.705324, current state activating+remapped, last acting [14,89,114,65,50,47,108,151,24,124]
pg 4.fd5 is stuck unclean for 183022.405365, current state unknown, last acting []
pg 4.fd6 is stuck undersized for 7369.422898, current state activating+undersized+degraded+remapped, last acting [20,2147483647,109,131,156,55,68,146,40,94]
pg 4.fd7 is stuck unclean for 471269.398287, current state activating+remapped, last acting [27,35,75,140,98,87,120,12,55,183]
pg 4.fd8 is stuck undersized for 9347.658890, current state activating+undersized+degraded+remapped, last acting [107,92,118,68,156,2147483647,54,139,136,22]
pg 4.fd9 is stuck unclean for 471269.891967, current state activating+remapped, last acting [41,182,167,14,143,71,175,120,92,22]
pg 4.fda is stuck undersized for 50079.828926, current state activating+undersized+degraded+remapped, last acting [131,21,82,162,78,94,60,112,156,2147483647]
pg 4.fdb is stuck unclean for 471262.448648, current state activating+remapped, last acting [33,81,176,66,40,124,120,57,149,9]
pg 4.fdc is stuck undersized for 50229.709062, current state activating+undersized+degraded+remapped, last acting [134,61,16,121,47,68,177,23,2147483647,85]
pg 4.fdd is stuck unclean for 471262.035474, current state activating+remapped, last acting [174,66,167,188,19,135,153,105,15,42]
pg 4.fde is stuck unclean for 471269.941065, current state activating+remapped, last acting [79,176,35,155,65,146,181,161,120,59]
pg 4.fdf is stuck undersized for 50229.509559, current state activating+undersized+degraded+remapped, last acting [105,119,49,7,23,171,79,42,2147483647,182]
pg 4.fe0 is stuck unclean for 471265.886110, current state activating+remapped, last acting [106,81,181,178,192,31,49,171,48,8]
pg 4.fe1 is stuck unclean for 183022.405365, current state unknown, last acting []
pg 4.fe2 is stuck undersized for 8532.475521, current state activating+undersized+degraded+remapped, last acting [121,5,167,187,177,33,169,2147483647,127,190]
pg 4.fe3 is stuck unclean for 471264.386079, current state activating+remapped, last acting [191,42,23,81,114,64,56,12,103,127]
pg 4.fe4 is stuck undersized for 50226.833549, current state activating+undersized+degraded+remapped, last acting [48,135,94,11,188,2147483647,33,171,80,60]
pg 4.fe5 is stuck undersized for 8532.522184, current state activating+undersized+degraded+remapped, last acting [9,67,2147483647,193,30,53,94,83,118,132]
pg 4.fe6 is stuck undersized for 50229.716524, current state activating+undersized+degraded+remapped, last acting [154,74,53,29,124,164,2147483647,178,105,85]
pg 4.fe7 is stuck undersized for 50226.866165, current state activating+undersized+degraded+remapped, last acting [2147483647,119,62,138,4,95,71,81,160,35]
pg 4.fe8 is stuck undersized for 50229.690669, current state activating+undersized+degraded+remapped, last acting [19,183,10,99,76,111,34,2147483647,90,166]
pg 4.fe9 is stuck undersized for 50232.722444, current state activating+undersized+degraded+remapped, last acting [160,71,39,128,109,49,156,91,106,2147483647]
pg 4.fea is stuck undersized for 7369.328946, current state activating+undersized+degraded+remapped, last acting [160,72,62,6,96,117,147,44,2147483647,135]
pg 4.feb is stuck undersized for 50229.691912, current state activating+undersized+degraded+remapped, last acting [138,169,89,2147483647,36,156,95,29,63,121]
pg 4.fec is stuck undersized for 50229.670411, current state activating+undersized+degraded+remapped, last acting [90,125,16,74,123,168,2147483647,45,186,20]
pg 4.fed is stuck undersized for 50233.594733, current state activating+undersized+degraded+remapped, last acting [5,171,22,189,180,34,186,2147483647,187,52]
pg 4.fee is stuck unclean for 471270.522871, current state activating+remapped, last acting [152,39,125,168,186,114,22,8,184,174]
pg 4.fef is stuck unclean for 471269.694096, current state activating+remapped, last acting [22,52,89,138,151,111,78,34,103,157]
pg 4.ff0 is stuck undersized for 50226.870743, current state activating+undersized+degraded+remapped, last acting [37,87,25,181,14,102,70,57,119,2147483647]
pg 4.ff1 is stuck undersized for 50226.959328, current state activating+undersized+degraded+remapped, last acting [31,63,165,122,157,172,125,74,175,2147483647]
pg 4.ff2 is stuck undersized for 50232.083606, current state activating+undersized+degraded+remapped, last acting [16,91,177,56,113,158,2147483647,65,38,135]
pg 4.ff3 is stuck undersized for 50229.719102, current state activating+undersized+degraded+remapped, last acting [74,100,57,188,80,132,44,28,2147483647,157]
pg 4.ff4 is stuck undersized for 50229.668648, current state activating+undersized+degraded+remapped, last acting [62,2147483647,42,120,106,92,28,155,75,181]
pg 4.ff5 is stuck unclean for 471264.473844, current state activating+remapped, last acting [89,148,119,156,163,106,32,171,130,53]
pg 4.ff6 is stuck undersized for 8532.558304, current state activating+undersized+degraded+remapped, last acting [7,56,66,178,190,94,2147483647,84,159,183]
pg 4.ff7 is stuck undersized for 50232.817408, current state activating+undersized+degraded+remapped, last acting [176,62,31,15,87,126,65,180,2147483647,47]
pg 4.ff8 is stuck undersized for 50229.681426, current state activating+undersized+degraded+remapped, last acting [173,185,34,21,184,14,120,2147483647,175,133]
pg 4.ff9 is stuck undersized for 7369.455991, current state activating+undersized+degraded+remapped, last acting [188,68,29,125,102,143,2147483647,45,156,55]
pg 4.ffa is stuck unclean for 471264.413388, current state activating+remapped, last acting [139,173,25,135,49,154,67,164,179,101]
pg 4.ffb is stuck unclean for 471270.205758, current state activating+remapped, last acting [83,47,182,113,59,11,103,27,184,147]
pg 4.ffc is stuck undersized for 50232.323347, current state activating+undersized+degraded+remapped, last acting [157,38,2147483647,172,121,161,104,73,61,130]
pg 4.ffd is stuck unclean for 471270.391099, current state activating+remapped, last acting [48,161,172,154,72,135,119,146,103,50]
pg 4.ffe is stuck unclean for 471268.499181, current state activating+remapped, last acting [156,175,33,169,135,85,165,55,148,178]
pg 4.fff is stuck undersized for 50226.310909, current state activating+undersized+degraded+remapped, last acting [44,12,185,125,69,29,119,102,81,2147483647]
REQUEST_STUCK 1 stuck requests are blocked > 4096 sec
1 ops are blocked > 8388.61 sec
osd.46 has stuck requests > 8388.61 sec