I first became suspicious about Milford's metronome markings when I was learning his Diversions for piano. The numbers duplicated Italian terms, and my instinct was that the latter were more reliable than the former. Some of the metronome markings were fast enough to render the pieces unplayable, it seemed to me.
Then I came across this letter cited In Ian Copley's book, from Milford to Edward Bradby, dated 28.04.1931, written after a concert of his own works with himself conducting.
"...It is quite true of course that I nearly always take my things too fast, and I was very interested in what you said about pace, which had not occurred to me before (I mean the part about vertical structure and the more contrapuntal movements), and I think this is undoubtedly true...
"It was very interesting going through 'A Prophet in the land' with Kennedy Scott. I had already metronomed it, and in almost every case he said I had put much too fast a speed, so I took it home and I must confess that in nearly every case I came to the conclusion that he was right, and I re-metronomed the whole thing."
So it seems to me that Milford's metronome markings, like those of Schumann and some other composers, should not be taken rigidly.